2024-01-21T19:46:45.968 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T19:46:45.968 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071 --verbose --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology --name yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/orig.config.yaml 2024-01-21T19:46:46.014 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T19:46:46.102 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071 branch: wip-yuri4-testing-2024-01-18-1257-pacific description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7525071' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-01-18-1257-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-01-18-1257-pacific sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9457 sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri4-testing-2024-01-18-1257-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec targets: smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJW5n2c9TyhtohRmtZgkhPVMm5LGgNPhxEtr2nFYL00v9GCcP28JflQFozomYWCYHu4RaVmNbH+VVgYG6DCzMg0= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRPqxZJRPPOqbJwp7GTJTBwgO6jxWOJIP+JCFihE9/i6l3IzUlgxoW4pzVtvM67bT9cuerqEvK/kb266Kd38vg= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-21_16:37:18 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-21T19:46:46.102 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa; will attempt to use it 2024-01-21T19:46:46.102 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks 2024-01-21T19:46:46.102 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T19:46:46.104 INFO:teuthology.task.internal:Checking packages... 2024-01-21T19:46:46.126 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec' 2024-01-21T19:46:46.126 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T19:46:46.126 INFO:teuthology.packaging:ref: None 2024-01-21T19:46:46.126 INFO:teuthology.packaging:tag: None 2024-01-21T19:46:46.126 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T19:46:46.126 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:46:46.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-01-18-1257-pacific 2024-01-21T19:46:46.347 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.gea310589 2024-01-21T19:46:46.348 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T19:46:46.355 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T19:46:46.355 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T19:46:46.404 INFO:teuthology.task.internal:Saving configuration 2024-01-21T19:46:46.422 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T19:46:46.428 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T19:46:46.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:41:23.004969', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJW5n2c9TyhtohRmtZgkhPVMm5LGgNPhxEtr2nFYL00v9GCcP28JflQFozomYWCYHu4RaVmNbH+VVgYG6DCzMg0='} 2024-01-21T19:46:46.474 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:41:23.006297', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRPqxZJRPPOqbJwp7GTJTBwgO6jxWOJIP+JCFihE9/i6l3IzUlgxoW4pzVtvM67bT9cuerqEvK/kb266Kd38vg='} 2024-01-21T19:46:46.474 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T19:46:46.480 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-21T19:46:46.480 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-21T19:46:46.480 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T19:46:46.578 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff43f647160>, signals=[15]) 2024-01-21T19:46:46.579 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T19:46:46.586 INFO:teuthology.task.internal:Opening connections... 2024-01-21T19:46:46.587 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-01-21T19:46:46.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:46:46.664 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-01-21T19:46:46.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:46:46.738 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T19:46:46.744 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-01-21T19:46:46.767 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-01-21T19:46:46.767 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T19:46:46.827 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T19:46:46.828 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-01-21T19:46:46.850 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-01-21T19:46:46.872 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-01-21T19:46:46.873 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-01-21T19:46:46.933 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-01-21T19:46:46.933 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 2024-01-21T19:46:46.933 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-01-21T19:46:46.933 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-21T19:46:46.934 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T19:46:46.934 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-01-21T19:46:46.957 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T19:46:46.966 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T19:46:47.013 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T19:46:47.013 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-01-21T19:46:47.016 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-01-21T19:46:47.037 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T19:46:47.047 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T19:46:47.047 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-01-21T19:46:47.077 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-01-21T19:46:47.104 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T19:46:47.158 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T19:46:47.172 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T19:46:47.172 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T19:46:47.172 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T19:46:47.173 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T19:46:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:46:47.193 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-01-21T19:46:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:46:47.194 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-01-21T19:46:47.254 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-489.el8.x86_64 2024-01-21T19:46:47.255 INFO:teuthology.task.kernel:Running kernel on smithi153: 4.18.0-489.el8.x86_64 2024-01-21T19:46:47.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-01-21T19:46:47.256 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-489.el8.x86_64 2024-01-21T19:46:47.256 INFO:teuthology.task.kernel:Running kernel on smithi119: 4.18.0-489.el8.x86_64 2024-01-21T19:46:47.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-01-21T19:46:50.862 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - AppStream 22 MB/s | 26 MB 00:01 2024-01-21T19:46:51.119 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - AppStream 21 MB/s | 26 MB 00:01 2024-01-21T19:46:56.312 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-01-21T19:46:56.460 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-01-21T19:46:58.544 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras 36 kB/s | 18 kB 00:00 2024-01-21T19:46:58.554 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras 27 kB/s | 18 kB 00:00 2024-01-21T19:46:58.897 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras common packages 30 kB/s | 6.9 kB 00:00 2024-01-21T19:46:59.036 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-21T19:46:59.231 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-01-21T19:46:59.430 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-01-21T19:47:01.497 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2024-01-21T19:47:01.624 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 16 MB 00:02 2024-01-21T19:47:05.284 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 440 kB/s | 24 kB 00:00 2024-01-21T19:47:05.398 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 536 kB/s | 24 kB 00:00 2024-01-21T19:47:08.160 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T19:47:08.160 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T19:47:08.263 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T19:47:08.264 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout:Install 3 Packages 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 89 M 2024-01-21T19:47:08.265 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 96 M 2024-01-21T19:47:08.266 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:47:08.287 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-21T19:47:08.287 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-21T19:47:08.428 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:47:08.429 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout:Install 3 Packages 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 89 M 2024-01-21T19:47:08.430 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 96 M 2024-01-21T19:47:08.431 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:47:09.140 INFO:teuthology.orchestra.run.smithi119.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2024-01-21T19:47:09.700 INFO:teuthology.orchestra.run.smithi153.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 8.3 MB/s | 10 MB 00:01 2024-01-21T19:47:10.565 INFO:teuthology.orchestra.run.smithi119.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-01-21T19:47:10.958 INFO:teuthology.orchestra.run.smithi153.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 14 MB/s | 36 MB 00:02 2024-01-21T19:47:11.065 INFO:teuthology.orchestra.run.smithi119.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2024-01-21T19:47:11.066 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:11.066 INFO:teuthology.orchestra.run.smithi119.stdout:Total 32 MB/s | 89 MB 00:02 2024-01-21T19:47:11.433 INFO:teuthology.orchestra.run.smithi153.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2024-01-21T19:47:11.434 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:47:11.434 INFO:teuthology.orchestra.run.smithi153.stdout:Total 30 MB/s | 89 MB 00:03 2024-01-21T19:47:11.749 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:47:12.120 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:47:12.160 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:47:12.160 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:47:12.546 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:47:12.547 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:47:13.394 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:47:13.415 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:47:13.767 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:47:13.791 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:47:16.155 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:47:16.530 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:47:16.657 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:47:17.083 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:47:20.203 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:47:20.616 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:47:20.639 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:47:20.990 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:47:24.065 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:47:24.336 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:24.420 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:47:24.687 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.099 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.121 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.464 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:47:46.465 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:47:46.530 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.554 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.906 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.906 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-21T19:47:46.906 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-21T19:47:46.921 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:46.922 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:47:46.922 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:47:46.922 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T19:47:46.922 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T19:47:46.922 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:47:46.922 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:47:47.086 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:47:47.302 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:47:47.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.302 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.302 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T19:47:47.302 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T19:47:47.302 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-21T19:47:47.302 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T19:47:47.303 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:47:47.354 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-21T19:47:47.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:47:47.355 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:47:47.355 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-21T19:47:47.355 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-21T19:47:47.355 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:47:47.355 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:47:47.482 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:47:47.512 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:47:47.512 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-01-21T19:47:47.546 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.02 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 129.el8 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Group : System Environment/Base 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 9265586 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Relocations : (not relocatable) 2024-01-21T19:47:47.547 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : CentOS BuildSystem 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:47:47.548 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T19:47:47.549 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T19:47:47.549 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T19:47:47.740 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:47:47.740 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.740 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.740 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T19:47:47.740 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T19:47:47.740 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-21T19:47:47.740 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T19:47:47.741 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T19:47:47.951 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-21T19:47:47.952 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-01-21T19:47:47.985 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-01-21T19:47:47.985 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-01-21T19:47:47.985 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.02 2024-01-21T19:47:47.985 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 129.el8 2024-01-21T19:47:47.985 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Group : System Environment/Base 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 9265586 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Relocations : (not relocatable) 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : CentOS BuildSystem 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-01-21T19:47:47.986 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:47:47.987 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T19:47:47.987 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T19:47:47.987 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T19:47:47.987 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-01-21T19:47:47.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:47:47.987 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T19:47:47.988 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T19:47:47.988 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 4.18.0-535.el8.x86_64 2024-01-21T19:47:47.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T19:47:49.365 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-01-21T19:47:49.816 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-01-21T19:47:51.614 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-01-21T19:47:51.616 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-01-21T19:47:51.642 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.87skVvvBx5 2024-01-21T19:47:51.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.87skVvvBx5 2024-01-21T19:47:51.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.87skVvvBx5 2024-01-21T19:47:51.884 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.87skVvvBx5 is 6KB 2024-01-21T19:47:51.907 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.87skVvvBx5 2024-01-21T19:47:51.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T19:47:51.999 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T19:47:51.999 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T19:47:51.999 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T19:47:51.999 INFO:teuthology.orchestra.run.smithi119.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T19:47:52.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T19:47:52.088 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-01-21T19:47:52.089 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-01-21T19:47:52.111 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.DBBKr65E71 2024-01-21T19:47:52.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DBBKr65E71 2024-01-21T19:47:52.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-01-21T19:47:52.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.DBBKr65E71 2024-01-21T19:47:52.328 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.DBBKr65E71 is 6KB 2024-01-21T19:47:52.375 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.DBBKr65E71 2024-01-21T19:47:52.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T19:47:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-21T19:47:52.469 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-21T19:47:52.470 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-21T19:47:52.470 INFO:teuthology.orchestra.run.smithi153.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-21T19:47:52.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-21T19:47:52.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-01-21T19:48:22.176 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-21T19:48:22.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-01-21T19:48:22.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:48:22.635 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-21T19:48:22.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-01-21T19:48:22.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:48:57.280 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-01-21T19:49:06.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-01-21T19:49:06.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:49:22.195 DEBUG:teuthology.orchestra.remote:timed out 2024-01-21T19:49:31.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-01-21T19:49:31.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:49:31.489 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-01-21T19:49:31.783 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-01-21T19:49:31.783 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-21T19:49:31.784 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-01-21T19:49:31.800 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-535.el8.x86_64 2024-01-21T19:49:31.800 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T19:49:31.800 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T19:49:31.801 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T19:49:32.801 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-21T19:49:32.802 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T19:49:32.896 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-01-21T19:49:32.910 DEBUG:teuthology.parallel:result is None 2024-01-21T19:49:38.010 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-01-21T19:49:38.342 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-01-21T19:49:38.342 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-21T19:49:38.342 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-01-21T19:49:38.361 INFO:teuthology.orchestra.run.smithi153.stdout:4.18.0-535.el8.x86_64 2024-01-21T19:49:38.361 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-21T19:49:38.361 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T19:49:38.362 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T19:49:39.362 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-21T19:49:39.363 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T19:49:39.468 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-01-21T19:49:39.486 DEBUG:teuthology.parallel:result is None 2024-01-21T19:49:39.486 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T19:49:39.493 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T19:49:39.494 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T19:49:39.497 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T19:49:39.517 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T19:49:39.534 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T19:49:39.567 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T19:49:39.568 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T19:49:39.570 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T19:49:39.610 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T19:49:39.617 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T19:49:39.617 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T19:49:39.649 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T19:49:39.681 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:49:39.691 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:49:39.695 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:49:39.706 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T19:49:39.707 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T19:49:39.714 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T19:49:39.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T19:49:39.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T19:49:39.766 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T19:49:39.776 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T19:49:39.777 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T19:49:39.806 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T19:49:39.824 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:49:39.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:49:39.962 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:49:40.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:49:40.089 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:49:40.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T19:49:40.156 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:49:40.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T19:49:40.251 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:49:40.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T19:49:40.377 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T19:49:40.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T19:49:40.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-01-21T19:49:40.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-01-21T19:49:40.516 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:49:40.575 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T19:49:40.851 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T19:49:40.859 INFO:teuthology.task.internal:Starting timer... 2024-01-21T19:49:40.859 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T19:49:40.899 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T19:49:40.977 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-21T19:49:40.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-01-21T19:49:41.065 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-01-21T19:49:41.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-01-21T19:49:41.156 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-01-21T19:49:41.157 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T19:49:41.158 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-01-21T19:49:41.219 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-01-21T19:49:41.220 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-01-21T19:49:41.241 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-01-21T19:49:41.241 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-01-21T19:49:41.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:49:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:49:41.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T19:49:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T19:49:41.308 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T19:49:41.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-01-21T19:49:41.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-01-21T19:49:41.376 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T19:49:41.386 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-21T19:49:41.391 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-21T19:49:41.498 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T19:49:41.512 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T19:49:41.513 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T19:55:25.771 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-01-21T19:55:25.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-01-21T19:55:25.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:55:25.856 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-01-21T19:55:25.921 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-01-21T19:55:25.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-01-21T19:55:25.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T19:55:25.997 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-01-21T19:55:26.066 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-01-21T19:55:26.066 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T19:55:26.076 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T19:55:26.076 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T19:55:26.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:55:26.079 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T19:55:26.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T19:55:26.108 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T19:55:26.122 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T19:55:26.137 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T19:55:26.150 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T19:55:26.189 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-01-21T19:55:26.200 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-01-21T19:55:26.214 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T19:55:26.224 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-01-21T19:55:26.227 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T19:55:26.234 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-01-21T19:55:26.248 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T19:55:26.261 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T19:55:26.302 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2024-01-21T19:55:26.305 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:55:26.305 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-01-21T19:55:26.305 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.305 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.306 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.306 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2024-01-21T19:55:26.350 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T19:55:26.350 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-01-21T19:55:26.350 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.350 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.350 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.350 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T19:55:26.351 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T19:55:26.360 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T19:55:26.361 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi119.front.sepia.ceph.com 2024-01-21T19:55:26.361 DEBUG:teuthology.orchestra.run.smithi119:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:55:26.361 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi153.front.sepia.ceph.com 2024-01-21T19:55:26.361 DEBUG:teuthology.orchestra.run.smithi153:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:55:27.112 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:29 ago on Sun 21 Jan 2024 07:53:57 PM UTC. 2024-01-21T19:55:27.152 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:19 ago on Sun 21 Jan 2024 07:54:07 PM UTC. 2024-01-21T19:55:29.015 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:55:29.015 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout:Resetting modules: 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout: container-tools 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:55:29.016 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:29.073 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout:Resetting modules: 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: container-tools 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:55:29.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:29.333 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:55:29.335 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:55:30.153 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 07:54:07 PM UTC. 2024-01-21T19:55:30.163 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:33 ago on Sun 21 Jan 2024 07:53:57 PM UTC. 2024-01-21T19:55:32.084 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:55:32.096 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout:Installing group/module packages: 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T19:55:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout:Installing module profiles: 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: container-tools/common 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling module streams: 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: container-tools rhel8 2024-01-21T19:55:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 27 M 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 89 M 2024-01-21T19:55:32.120 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:55:32.131 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:55:32.131 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:Installing group/module packages: 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-21T19:55:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Installing module profiles: 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: container-tools/common 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling module streams: 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: container-tools rhel8 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-01-21T19:55:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:32.134 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 27 M 2024-01-21T19:55:32.134 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 89 M 2024-01-21T19:55:32.134 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:55:32.559 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.6 MB/s | 1.0 MB 00:00 2024-01-21T19:55:32.575 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.7 MB/s | 750 kB 00:00 2024-01-21T19:55:32.585 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): cockpit-podman-78-1.module_el8+712+4cd1 1.8 MB/s | 750 kB 00:00 2024-01-21T19:55:32.601 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 902 kB/s | 38 kB 00:00 2024-01-21T19:55:32.609 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.3 MB/s | 1.0 MB 00:00 2024-01-21T19:55:32.634 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 646 kB/s | 38 kB 00:00 2024-01-21T19:55:32.643 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2024-01-21T19:55:32.668 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2024-01-21T19:55:32.702 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2024-01-21T19:55:32.726 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2024-01-21T19:55:32.744 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 594 kB/s | 25 kB 00:00 2024-01-21T19:55:32.794 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-01-21T19:55:32.799 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 341 kB/s | 25 kB 00:00 2024-01-21T19:55:32.860 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.5 MB/s | 94 kB 00:00 2024-01-21T19:55:32.970 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 11 MB/s | 3.9 MB 00:00 2024-01-21T19:55:33.111 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 8.2 MB/s | 3.9 MB 00:00 2024-01-21T19:55:33.161 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 13 MB/s | 2.5 MB 00:00 2024-01-21T19:55:33.220 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 905 kB/s | 53 kB 00:00 2024-01-21T19:55:33.270 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 816 kB/s | 41 kB 00:00 2024-01-21T19:55:33.328 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 12 MB/s | 2.5 MB 00:00 2024-01-21T19:55:33.378 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.0 MB/s | 53 kB 00:00 2024-01-21T19:55:33.428 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 819 kB/s | 41 kB 00:00 2024-01-21T19:55:33.429 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 7.7 MB/s | 9.6 MB 00:01 2024-01-21T19:55:33.538 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 11 MB/s | 8.1 MB 00:00 2024-01-21T19:55:33.538 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:33.538 INFO:teuthology.orchestra.run.smithi119.stdout:Total 19 MB/s | 27 MB 00:01 2024-01-21T19:55:33.579 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 7.5 MB/s | 9.6 MB 00:01 2024-01-21T19:55:33.787 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 8.8 MB/s | 8.1 MB 00:00 2024-01-21T19:55:33.788 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:55:33.788 INFO:teuthology.orchestra.run.smithi153.stdout:Total 16 MB/s | 27 MB 00:01 2024-01-21T19:55:33.800 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:55:33.820 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:55:33.821 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:55:34.053 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:55:34.057 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:55:34.061 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:55:34.074 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:55:34.074 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:55:34.334 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:55:34.338 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:55:34.711 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:55:34.943 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T19:55:35.039 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:55:35.105 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T19:55:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-21T19:55:35.308 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T19:55:35.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-21T19:55:35.530 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:55:35.652 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-21T19:55:35.859 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:55:36.075 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T19:55:36.420 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-21T19:55:37.028 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:55:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T19:55:37.407 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:55:37.448 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T19:55:37.654 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-21T19:55:37.758 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T19:55:37.859 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-21T19:55:38.144 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-21T19:55:38.406 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T19:55:38.534 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-21T19:55:39.271 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:55:39.422 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:55:39.484 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T19:55:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-21T19:55:39.670 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:55:39.805 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:55:40.070 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:55:40.071 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-21T19:55:40.243 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-21T19:55:40.244 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T19:55:40.546 INFO:teuthology.orchestra.run.smithi119.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.547 INFO:teuthology.orchestra.run.smithi119.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.547 INFO:teuthology.orchestra.run.smithi119.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T19:55:40.547 INFO:teuthology.orchestra.run.smithi119.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T19:55:40.547 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:55:40.547 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:55:40.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-21T19:55:40.795 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:40.795 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:55:40.795 INFO:teuthology.orchestra.run.smithi153.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.795 INFO:teuthology.orchestra.run.smithi153.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:55:40.796 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:55:40.806 DEBUG:teuthology.parallel:result is None 2024-01-21T19:55:41.131 DEBUG:teuthology.parallel:result is None 2024-01-21T19:55:41.131 INFO:teuthology.run_tasks:Running task pexec... 2024-01-21T19:55:41.167 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-21T19:55:41.167 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi119.front.sepia.ceph.com 2024-01-21T19:55:41.167 DEBUG:teuthology.orchestra.run.smithi119:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:55:41.168 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi153.front.sepia.ceph.com 2024-01-21T19:55:41.168 DEBUG:teuthology.orchestra.run.smithi153:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-21T19:55:41.361 DEBUG:teuthology.parallel:result is None 2024-01-21T19:55:41.386 DEBUG:teuthology.parallel:result is None 2024-01-21T19:55:41.386 INFO:teuthology.run_tasks:Running task install... 2024-01-21T19:55:41.402 DEBUG:teuthology.task.install:project ceph 2024-01-21T19:55:41.402 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'}} 2024-01-21T19:55:41.402 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T19:55:41.403 INFO:teuthology.task.install:Using flavor: default 2024-01-21T19:55:41.414 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-21T19:55:41.414 INFO:teuthology.task.install:extra packages: [] 2024-01-21T19:55:41.415 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T19:55:41.415 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:55:41.415 INFO:teuthology.packaging:ref: None 2024-01-21T19:55:41.415 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:55:41.416 INFO:teuthology.packaging:branch: None 2024-01-21T19:55:41.416 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:55:41.596 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:55:41.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:55:41.599 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-21T19:55:41.599 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:55:41.599 INFO:teuthology.packaging:ref: None 2024-01-21T19:55:41.599 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:55:41.599 INFO:teuthology.packaging:branch: None 2024-01-21T19:55:41.599 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:55:41.600 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:55:41.739 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T19:55:41.739 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T19:55:41.752 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-21T19:55:41.752 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-21T19:55:41.784 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T19:55:41.785 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:55:41.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T19:55:41.790 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T19:55:41.791 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T19:55:41.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T19:55:41.814 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T19:55:41.814 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:55:41.814 INFO:teuthology.packaging:ref: None 2024-01-21T19:55:41.814 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:55:41.814 INFO:teuthology.packaging:branch: None 2024-01-21T19:55:41.814 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:55:41.814 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T19:55:41.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-21T19:55:41.818 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:55:41.818 INFO:teuthology.packaging:ref: None 2024-01-21T19:55:41.818 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:55:41.818 INFO:teuthology.packaging:branch: None 2024-01-21T19:55:41.818 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:55:41.818 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T19:55:41.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T19:55:41.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T19:55:41.958 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T19:55:41.962 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T19:55:42.024 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-01-21T19:55:42.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-01-21T19:55:42.029 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-01-21T19:55:42.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-01-21T19:55:42.284 INFO:teuthology.orchestra.run.smithi119.stdout:76 files removed 2024-01-21T19:55:42.305 INFO:teuthology.orchestra.run.smithi153.stdout:76 files removed 2024-01-21T19:55:42.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-01-21T19:55:42.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-01-21T19:55:42.788 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-01-21T19:55:42.829 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 841 kB/s | 74 kB 00:00 2024-01-21T19:55:42.896 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 220 kB/s | 15 kB 00:00 2024-01-21T19:55:42.938 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 213 kB/s | 15 kB 00:00 2024-01-21T19:55:42.997 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 25 kB/s | 1.7 kB 00:00 2024-01-21T19:55:43.031 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 27 kB/s | 1.7 kB 00:00 2024-01-21T19:55:44.465 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - AppStream 19 MB/s | 26 MB 00:01 2024-01-21T19:55:45.114 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - AppStream 13 MB/s | 26 MB 00:02 2024-01-21T19:55:49.824 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 10 MB 00:00 2024-01-21T19:55:50.473 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-01-21T19:55:52.116 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras 25 kB/s | 18 kB 00:00 2024-01-21T19:55:52.457 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2024-01-21T19:55:52.783 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-01-21T19:55:52.958 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-21T19:55:53.965 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 8 - PowerTools 4.3 MB/s | 4.7 MB 00:01 2024-01-21T19:55:54.276 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 8 - PowerTools 3.9 MB/s | 4.7 MB 00:01 2024-01-21T19:55:55.069 INFO:teuthology.orchestra.run.smithi153.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-01-21T19:55:55.286 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-01-21T19:55:56.895 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 16 MB 00:01 2024-01-21T19:55:57.015 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2024-01-21T19:56:00.670 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 445 kB/s | 24 kB 00:00 2024-01-21T19:56:00.811 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 434 kB/s | 24 kB 00:00 2024-01-21T19:56:04.141 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:56:04.144 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:04.144 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T19:56:04.145 INFO:teuthology.orchestra.run.smithi153.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T19:56:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout:Install 25 Packages 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-01-21T19:56:04.147 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:04.148 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 59 M 2024-01-21T19:56:04.148 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:56:04.247 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:56:04.250 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:04.250 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:56:04.250 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:04.250 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:56:04.250 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-21T19:56:04.250 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-21T19:56:04.251 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-21T19:56:04.252 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout:Install 25 Packages 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 59 M 2024-01-21T19:56:04.253 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:56:04.405 INFO:teuthology.orchestra.run.smithi153.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 41 MB/s | 6.0 MB 00:00 2024-01-21T19:56:04.422 INFO:teuthology.orchestra.run.smithi153.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-01-21T19:56:04.455 INFO:teuthology.orchestra.run.smithi153.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-01-21T19:56:04.489 INFO:teuthology.orchestra.run.smithi153.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T19:56:04.581 INFO:teuthology.orchestra.run.smithi153.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-01-21T19:56:04.597 INFO:teuthology.orchestra.run.smithi153.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.7 MB/s | 44 kB 00:00 2024-01-21T19:56:04.623 INFO:teuthology.orchestra.run.smithi153.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T19:56:04.672 INFO:teuthology.orchestra.run.smithi153.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.2 MB/s | 213 kB 00:00 2024-01-21T19:56:04.707 INFO:teuthology.orchestra.run.smithi153.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T19:56:04.740 INFO:teuthology.orchestra.run.smithi153.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-01-21T19:56:04.766 INFO:teuthology.orchestra.run.smithi153.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T19:56:04.900 INFO:teuthology.orchestra.run.smithi153.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-21T19:56:05.062 INFO:teuthology.orchestra.run.smithi119.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2024-01-21T19:56:05.087 INFO:teuthology.orchestra.run.smithi119.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-01-21T19:56:05.121 INFO:teuthology.orchestra.run.smithi119.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-21T19:56:05.154 INFO:teuthology.orchestra.run.smithi119.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-21T19:56:05.183 INFO:teuthology.orchestra.run.smithi153.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-21T19:56:05.209 INFO:teuthology.orchestra.run.smithi153.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 328 kB/s | 145 kB 00:00 2024-01-21T19:56:05.246 INFO:teuthology.orchestra.run.smithi119.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-01-21T19:56:05.259 INFO:teuthology.orchestra.run.smithi153.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 880 kB/s | 44 kB 00:00 2024-01-21T19:56:05.263 INFO:teuthology.orchestra.run.smithi119.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-21T19:56:05.289 INFO:teuthology.orchestra.run.smithi119.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-21T19:56:05.292 INFO:teuthology.orchestra.run.smithi153.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-01-21T19:56:05.314 INFO:teuthology.orchestra.run.smithi119.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-21T19:56:05.348 INFO:teuthology.orchestra.run.smithi119.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-21T19:56:05.374 INFO:teuthology.orchestra.run.smithi119.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-01-21T19:56:05.384 INFO:teuthology.orchestra.run.smithi153.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2024-01-21T19:56:05.398 INFO:teuthology.orchestra.run.smithi119.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-21T19:56:05.418 INFO:teuthology.orchestra.run.smithi153.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 265 kB/s | 42 kB 00:00 2024-01-21T19:56:05.452 INFO:teuthology.orchestra.run.smithi153.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2024-01-21T19:56:05.527 INFO:teuthology.orchestra.run.smithi153.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 1.0 MB/s | 116 kB 00:00 2024-01-21T19:56:05.532 INFO:teuthology.orchestra.run.smithi119.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-21T19:56:05.572 INFO:teuthology.orchestra.run.smithi153.stdout:(21/27): ndctl-libs-71.1-7.el8.x86_64.rpm 436 kB/s | 80 kB 00:00 2024-01-21T19:56:05.628 INFO:teuthology.orchestra.run.smithi153.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2024-01-21T19:56:05.661 INFO:teuthology.orchestra.run.smithi153.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-01-21T19:56:05.687 INFO:teuthology.orchestra.run.smithi153.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 501 kB/s | 59 kB 00:00 2024-01-21T19:56:05.712 INFO:teuthology.orchestra.run.smithi153.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 904 kB/s | 75 kB 00:00 2024-01-21T19:56:05.795 INFO:teuthology.orchestra.run.smithi153.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-01-21T19:56:05.799 INFO:teuthology.orchestra.run.smithi119.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-21T19:56:05.824 INFO:teuthology.orchestra.run.smithi119.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 341 kB/s | 145 kB 00:00 2024-01-21T19:56:05.862 INFO:teuthology.orchestra.run.smithi153.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-01-21T19:56:05.864 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:05.864 INFO:teuthology.orchestra.run.smithi153.stdout:Total 34 MB/s | 59 MB 00:01 2024-01-21T19:56:05.883 INFO:teuthology.orchestra.run.smithi119.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 752 kB/s | 44 kB 00:00 2024-01-21T19:56:05.917 INFO:teuthology.orchestra.run.smithi119.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 677 kB 00:00 2024-01-21T19:56:05.930 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:56:05.979 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:56:05.979 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:56:06.000 INFO:teuthology.orchestra.run.smithi119.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2024-01-21T19:56:06.042 INFO:teuthology.orchestra.run.smithi119.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 266 kB/s | 42 kB 00:00 2024-01-21T19:56:06.076 INFO:teuthology.orchestra.run.smithi119.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2024-01-21T19:56:06.193 INFO:teuthology.orchestra.run.smithi119.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 417 kB/s | 80 kB 00:00 2024-01-21T19:56:06.322 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:56:06.325 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:56:06.585 INFO:teuthology.orchestra.run.smithi119.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 214 kB/s | 116 kB 00:00 2024-01-21T19:56:06.718 INFO:teuthology.orchestra.run.smithi119.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 477 kB/s | 306 kB 00:00 2024-01-21T19:56:06.744 INFO:teuthology.orchestra.run.smithi119.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 371 kB/s | 59 kB 00:00 2024-01-21T19:56:06.769 INFO:teuthology.orchestra.run.smithi119.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 314 kB/s | 181 kB 00:00 2024-01-21T19:56:06.853 INFO:teuthology.orchestra.run.smithi119.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-01-21T19:56:06.878 INFO:teuthology.orchestra.run.smithi119.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 472 kB/s | 75 kB 00:00 2024-01-21T19:56:06.936 INFO:teuthology.orchestra.run.smithi119.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-01-21T19:56:06.941 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:06.941 INFO:teuthology.orchestra.run.smithi119.stdout:Total 22 MB/s | 59 MB 00:02 2024-01-21T19:56:07.008 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:56:07.055 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:56:07.055 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:56:07.394 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:56:07.397 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:56:07.447 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:56:08.167 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T19:56:08.382 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T19:56:08.473 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:56:08.552 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:56:08.680 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-21T19:56:08.884 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-21T19:56:08.969 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:56:09.110 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:56:09.188 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:56:09.401 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:56:09.577 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:56:09.642 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:56:09.825 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:56:09.861 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T19:56:10.411 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T19:56:10.518 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-21T19:56:10.551 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:56:10.632 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:56:10.732 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:56:10.816 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:56:10.856 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T19:56:11.043 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:56:11.072 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:56:11.089 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:56:11.233 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:56:11.248 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:56:11.285 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:56:11.443 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:56:11.517 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-21T19:56:11.634 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:56:11.691 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:56:11.885 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:56:12.083 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-21T19:56:12.224 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:56:12.299 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:56:12.320 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:56:12.479 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-21T19:56:12.484 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:56:12.585 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:56:12.682 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:56:12.748 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-21T19:56:12.788 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T19:56:12.920 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:56:12.994 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T19:56:13.099 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:56:13.159 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:56:13.241 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:56:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:56:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-21T19:56:13.478 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:56:13.873 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T19:56:13.894 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:56:14.035 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T19:56:14.057 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:56:14.149 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-21T19:56:14.212 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T19:56:14.310 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-21T19:56:14.435 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T19:56:14.517 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-21T19:56:14.649 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:56:14.723 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-21T19:56:14.933 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-21T19:56:15.098 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-21T19:56:15.280 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-21T19:56:15.507 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-21T19:56:17.011 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:56:17.251 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:56:17.962 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:56:18.041 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:56:18.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:56:18.257 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:56:18.362 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:56:18.533 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:56:18.973 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-21T19:56:19.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:56:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-21T19:56:19.523 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /dev 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /run 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /dev 2024-01-21T19:56:37.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /run 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2024-01-21T19:56:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:39.058 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:56:39.158 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:56:39.158 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:56:39.365 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:56:39.365 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:56:40.072 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T19:56:40.073 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:56:40.074 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T19:56:40.074 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T19:56:40.081 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:56:40.140 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-21T19:56:40.140 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:56:40.330 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-21T19:56:40.330 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-21T19:56:41.003 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-21T19:56:41.004 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-21T19:56:41.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-21T19:56:41.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-21T19:56:41.005 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.121 INFO:teuthology.orchestra.run.smithi153.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.122 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:41.123 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:56:41.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-01-21T19:56:41.941 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: leveldb-1.22-1.el8.x86_64 2024-01-21T19:56:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-21T19:56:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.944 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.944 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-21T19:56:41.944 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:41.944 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:56:42.138 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:42 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:56:42.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-01-21T19:56:42.961 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:42 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:56:43.610 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:56:43.611 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:43.611 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:56:43.611 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-01-21T19:56:43.612 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:43.613 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 53 M 2024-01-21T19:56:43.613 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 229 M 2024-01-21T19:56:43.613 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:56:43.848 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 169 kB 00:00 2024-01-21T19:56:44.032 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-01-21T19:56:44.057 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 782 kB/s | 203 kB 00:00 2024-01-21T19:56:44.107 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T19:56:44.132 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.1 MB/s | 323 kB 00:00 2024-01-21T19:56:44.436 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:56:44.437 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:44.437 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-21T19:56:44.438 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-21T19:56:44.439 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-21T19:56:44.439 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:44.439 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:56:44.439 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:44.439 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-01-21T19:56:44.439 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:44.440 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 53 M 2024-01-21T19:56:44.440 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 229 M 2024-01-21T19:56:44.440 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:56:44.539 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2024-01-21T19:56:44.583 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2024-01-21T19:56:44.585 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:44.585 INFO:teuthology.orchestra.run.smithi153.stdout:Total 54 MB/s | 53 MB 00:00 2024-01-21T19:56:44.627 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:56:44.648 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:56:44.648 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:56:44.706 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 903 kB/s | 203 kB 00:00 2024-01-21T19:56:44.731 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 982 kB/s | 188 kB 00:00 2024-01-21T19:56:44.781 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-21T19:56:44.807 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.1 MB/s | 323 kB 00:00 2024-01-21T19:56:44.965 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:56:44.968 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:56:45.299 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 63 MB/s | 52 MB 00:00 2024-01-21T19:56:45.301 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:45.301 INFO:teuthology.orchestra.run.smithi119.stdout:Total 61 MB/s | 53 MB 00:00 2024-01-21T19:56:45.343 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:56:45.364 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:56:45.364 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:56:45.455 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:56:45.685 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:56:45.688 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:56:45.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T19:56:45.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T19:56:46.054 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:56:46.154 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:56:46.165 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:56:46.311 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T19:56:46.379 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-21T19:56:46.445 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:56:46.588 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-21T19:56:46.735 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:56:46.812 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-21T19:56:46.975 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-21T19:56:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:56:51.304 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:56:51.510 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:56:51.804 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:56:51.804 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T19:56:51.804 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T19:56:51.804 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T19:56:51.804 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T19:56:51.804 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T19:56:51.972 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-21T19:56:52.199 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:52.278 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:56:52.487 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-21T19:56:52.487 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-21T19:56:52.487 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-21T19:56:52.487 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-21T19:56:52.487 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-21T19:56:52.487 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-21T19:56:52.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-01-21T19:56:52.896 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-21T19:56:52.896 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:52.897 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:56:52.897 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-21T19:56:52.897 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-21T19:56:52.897 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-21T19:56:52.897 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:52.897 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:56:53.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-01-21T19:56:53.230 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:56:53.833 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:53 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:56:54.717 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:56:54.720 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:54.720 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T19:56:54.721 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T19:56:54.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout:Install 32 Packages 2024-01-21T19:56:54.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:56:54.724 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 M 2024-01-21T19:56:54.724 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 111 M 2024-01-21T19:56:54.724 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:56:55.314 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-21T19:56:55.317 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-21T19:56:55.318 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-21T19:56:55.319 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Install 32 Packages 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 31 M 2024-01-21T19:56:55.320 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 111 M 2024-01-21T19:56:55.321 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:56:55.359 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 100 kB/s | 6.3 kB 00:00 2024-01-21T19:56:55.437 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-01-21T19:56:55.493 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-01-21T19:56:55.527 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-01-21T19:56:55.627 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 19 MB/s | 5.0 MB 00:00 2024-01-21T19:56:55.634 INFO:teuthology.orchestra.run.smithi119.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 155 kB/s | 6.3 kB 00:00 2024-01-21T19:56:55.684 INFO:teuthology.orchestra.run.smithi119.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-01-21T19:56:55.735 INFO:teuthology.orchestra.run.smithi119.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-01-21T19:56:55.760 INFO:teuthology.orchestra.run.smithi119.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-21T19:56:55.844 INFO:teuthology.orchestra.run.smithi119.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2024-01-21T19:56:55.852 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-21T19:56:55.898 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 143 kB/s | 39 kB 00:00 2024-01-21T19:56:55.937 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 383 kB/s | 157 kB 00:00 2024-01-21T19:56:55.987 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 646 kB/s | 57 kB 00:00 2024-01-21T19:56:56.061 INFO:teuthology.orchestra.run.smithi119.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 48 MB/s | 18 MB 00:00 2024-01-21T19:56:56.062 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 494 kB/s | 103 kB 00:00 2024-01-21T19:56:56.086 INFO:teuthology.orchestra.run.smithi119.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 160 kB/s | 39 kB 00:00 2024-01-21T19:56:56.096 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.8 MB/s | 457 kB 00:00 2024-01-21T19:56:56.111 INFO:teuthology.orchestra.run.smithi119.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 447 kB/s | 157 kB 00:00 2024-01-21T19:56:56.179 INFO:teuthology.orchestra.run.smithi119.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 619 kB/s | 57 kB 00:00 2024-01-21T19:56:56.204 INFO:teuthology.orchestra.run.smithi119.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.9 MB/s | 457 kB 00:00 2024-01-21T19:56:56.229 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2024-01-21T19:56:56.245 INFO:teuthology.orchestra.run.smithi119.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 561 kB/s | 103 kB 00:00 2024-01-21T19:56:56.296 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 545 kB/s | 109 kB 00:00 2024-01-21T19:56:56.397 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.9 MB/s | 641 kB 00:00 2024-01-21T19:56:56.437 INFO:teuthology.orchestra.run.smithi119.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 962 kB/s | 249 kB 00:00 2024-01-21T19:56:56.455 INFO:teuthology.orchestra.run.smithi119.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 522 kB/s | 109 kB 00:00 2024-01-21T19:56:56.547 INFO:teuthology.orchestra.run.smithi119.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.8 MB/s | 641 kB 00:00 2024-01-21T19:56:56.714 INFO:teuthology.orchestra.run.smithi119.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 158 kB/s | 44 kB 00:00 2024-01-21T19:56:56.797 INFO:teuthology.orchestra.run.smithi119.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 541 kB/s | 185 kB 00:00 2024-01-21T19:56:56.872 INFO:teuthology.orchestra.run.smithi119.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-01-21T19:56:56.897 INFO:teuthology.orchestra.run.smithi119.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 492 kB/s | 173 kB 00:00 2024-01-21T19:56:56.948 INFO:teuthology.orchestra.run.smithi119.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2024-01-21T19:56:56.973 INFO:teuthology.orchestra.run.smithi119.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-01-21T19:56:56.998 INFO:teuthology.orchestra.run.smithi119.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 486 kB/s | 49 kB 00:00 2024-01-21T19:56:57.040 INFO:teuthology.orchestra.run.smithi119.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 636 kB/s | 59 kB 00:00 2024-01-21T19:56:57.082 INFO:teuthology.orchestra.run.smithi119.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2024-01-21T19:56:57.108 INFO:teuthology.orchestra.run.smithi119.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-01-21T19:56:57.133 INFO:teuthology.orchestra.run.smithi119.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-01-21T19:56:57.166 INFO:teuthology.orchestra.run.smithi119.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-01-21T19:56:57.191 INFO:teuthology.orchestra.run.smithi119.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2024-01-21T19:56:57.222 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 44 kB/s | 44 kB 00:00 2024-01-21T19:56:57.225 INFO:teuthology.orchestra.run.smithi119.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 299 kB/s | 27 kB 00:00 2024-01-21T19:56:57.317 INFO:teuthology.orchestra.run.smithi119.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2024-01-21T19:56:57.342 INFO:teuthology.orchestra.run.smithi119.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2024-01-21T19:56:57.367 INFO:teuthology.orchestra.run.smithi119.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 601 kB/s | 85 kB 00:00 2024-01-21T19:56:57.435 INFO:teuthology.orchestra.run.smithi119.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 195 kB/s | 23 kB 00:00 2024-01-21T19:56:57.440 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:57.440 INFO:teuthology.orchestra.run.smithi119.stdout:Total 15 MB/s | 31 MB 00:02 2024-01-21T19:56:57.519 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:56:57.557 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:56:57.558 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:56:57.805 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 123 kB/s | 185 kB 00:01 2024-01-21T19:56:57.881 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:56:57.881 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:56:57.906 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 114 kB/s | 173 kB 00:01 2024-01-21T19:56:58.047 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 46 kB/s | 11 kB 00:00 2024-01-21T19:56:58.156 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 70 kB/s | 18 kB 00:00 2024-01-21T19:56:58.248 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 375 kB/s | 384 kB 00:01 2024-01-21T19:56:58.323 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 177 kB/s | 49 kB 00:00 2024-01-21T19:56:58.449 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 201 kB/s | 59 kB 00:00 2024-01-21T19:56:58.574 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 62 kB/s | 16 kB 00:00 2024-01-21T19:56:58.599 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 806 kB/s | 283 kB 00:00 2024-01-21T19:56:58.716 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 70 kB/s | 19 kB 00:00 2024-01-21T19:56:58.849 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 87 kB/s | 24 kB 00:00 2024-01-21T19:56:58.874 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 106 kB/s | 29 kB 00:00 2024-01-21T19:56:58.974 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:56:59.008 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 94 kB/s | 27 kB 00:00 2024-01-21T19:56:59.180 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T19:56:59.242 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 684 kB/s | 251 kB 00:00 2024-01-21T19:56:59.350 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 510 kB/s | 255 kB 00:00 2024-01-21T19:56:59.375 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 232 kB/s | 85 kB 00:00 2024-01-21T19:56:59.382 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T19:56:59.526 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 81 kB/s | 23 kB 00:00 2024-01-21T19:56:59.530 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:56:59.530 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.4 MB/s | 31 MB 00:04 2024-01-21T19:56:59.607 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T19:56:59.611 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:56:59.650 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:56:59.650 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:56:59.797 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T19:56:59.971 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T19:56:59.972 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:56:59.974 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:00.150 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T19:57:00.319 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T19:57:00.488 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T19:57:00.671 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T19:57:00.899 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T19:57:01.095 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T19:57:01.106 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:01.307 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-21T19:57:01.344 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T19:57:01.536 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-21T19:57:01.695 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T19:57:01.787 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-21T19:57:01.952 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T19:57:02.017 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-21T19:57:02.107 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T19:57:02.198 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-21T19:57:02.339 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T19:57:02.364 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-21T19:57:02.500 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:57:02.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-21T19:57:02.691 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-21T19:57:02.790 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T19:57:02.885 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-21T19:57:03.013 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T19:57:03.098 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-21T19:57:03.192 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T19:57:03.301 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-21T19:57:03.405 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T19:57:03.569 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-21T19:57:03.648 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T19:57:03.880 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T19:57:03.957 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-21T19:57:04.108 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T19:57:04.189 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-21T19:57:04.377 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-21T19:57:04.417 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T19:57:04.617 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-21T19:57:04.704 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T19:57:04.794 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:57:05.087 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-21T19:57:05.089 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T19:57:05.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-21T19:57:05.319 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:57:05.464 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-21T19:57:05.655 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-21T19:57:05.937 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-21T19:57:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-21T19:57:06.403 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-21T19:57:06.759 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-21T19:57:06.849 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:57:07.035 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:57:07.071 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-21T19:57:07.458 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-21T19:57:07.466 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:57:07.678 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:57:07.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:57:07.906 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:57:08.074 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:57:08.091 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:57:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:57:08.632 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:57:08.632 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T19:57:08.632 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T19:57:08.633 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T19:57:08.634 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T19:57:08.635 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T19:57:09.208 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-21T19:57:09.369 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:57:09.564 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T19:57:09.564 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:09.564 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T19:57:09.565 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T19:57:09.566 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T19:57:09.567 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T19:57:09.567 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T19:57:09.567 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T19:57:09.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:09.567 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:09.802 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-21T19:57:09.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-01-21T19:57:10.044 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:57:10.267 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-21T19:57:10.467 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:57:10.484 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-21T19:57:10.556 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:10 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:10.616 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:57:11.024 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-21T19:57:11.024 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-21T19:57:11.024 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-21T19:57:11.025 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-21T19:57:11.026 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:11.915 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-21T19:57:11.916 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-2.6-1.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-21T19:57:11.917 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:11.918 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:11.963 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:12.008 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:12.010 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:57:12.010 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:12.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-01-21T19:57:12.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-01-21T19:57:12.852 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:12 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:12.914 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:12 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:14.335 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:14.335 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:14.335 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:57:14.336 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:14.337 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 71 k 2024-01-21T19:57:14.337 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 290 k 2024-01-21T19:57:14.337 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:57:14.357 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:14.382 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-21T19:57:14.382 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:14.382 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-01-21T19:57:14.383 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:57:14.389 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:57:14.390 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:57:14.391 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:57:14.394 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:57:14.403 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:14.405 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:57:14.405 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:14.440 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:57:14.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-01-21T19:57:14.936 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:15.138 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:15.238 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:15 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:15.363 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:15.719 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:15.720 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:15.720 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:15.720 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T19:57:15.720 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:15.720 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:15.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T19:57:16.667 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:16.714 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:57:16.715 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:16.716 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:57:16.716 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:16.716 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 71 k 2024-01-21T19:57:16.716 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 290 k 2024-01-21T19:57:16.716 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:57:16.747 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-16.2.4-0.el8.noarch.rpm 2.3 MB/s | 71 kB 00:00 2024-01-21T19:57:16.747 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:16.747 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.2 MB/s | 71 kB 00:00 2024-01-21T19:57:16.748 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:57:16.755 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:57:16.755 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:57:16.756 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:57:16.759 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:16.802 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:17.294 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:17.464 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:17.696 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:18.163 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:18.163 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:18.163 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:57:18.164 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:18.165 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 163 k 2024-01-21T19:57:18.165 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 587 k 2024-01-21T19:57:18.165 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:57:18.242 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.1 MB/s | 163 kB 00:00 2024-01-21T19:57:18.242 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:18.243 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.0 MB/s | 163 kB 00:00 2024-01-21T19:57:18.243 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:57:18.253 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:57:18.253 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:57:18.300 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:57:18.303 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:57:18.411 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:57:18.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-21T19:57:18.541 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:18.541 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:18.541 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-21T19:57:18.541 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:18.541 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:18.572 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:18.979 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:19.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T19:57:19.826 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:19 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:19.953 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:19.953 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:19.953 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:19.953 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:19.953 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:19.953 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:21.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-01-21T19:57:21.322 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:21.323 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:57:21.324 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:21.324 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:57:21.324 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:21.324 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 163 k 2024-01-21T19:57:21.324 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 587 k 2024-01-21T19:57:21.324 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:57:21.368 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.6 MB/s | 163 kB 00:00 2024-01-21T19:57:21.369 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:21.369 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.5 MB/s | 163 kB 00:00 2024-01-21T19:57:21.369 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:57:21.380 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:57:21.380 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:57:21.427 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:57:21.430 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:21.541 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:21.714 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:22.035 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:22 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:22.130 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:22.499 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:57:22.499 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:22.499 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:22.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-21T19:57:22.499 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:22.499 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:22.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-01-21T19:57:23.446 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:23 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:23.486 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:23.532 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:23.533 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:57:23.533 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:23.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T19:57:24.353 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:24 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:24.891 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:57:24.937 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:24.938 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:57:24.938 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:25.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T19:57:25.771 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:25 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:25.840 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:25.841 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:25.841 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:57:25.842 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:25.843 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-01-21T19:57:25.843 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:25.843 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.0 M 2024-01-21T19:57:25.843 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 15 M 2024-01-21T19:57:25.843 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:57:26.474 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 211 kB/s | 24 kB 00:00 2024-01-21T19:57:26.499 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 64 kB/s | 8.9 kB 00:00 2024-01-21T19:57:26.557 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 13 MB/s | 2.7 MB 00:00 2024-01-21T19:57:26.633 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 252 kB/s | 34 kB 00:00 2024-01-21T19:57:26.675 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 212 kB/s | 43 kB 00:00 2024-01-21T19:57:26.700 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2024-01-21T19:57:26.702 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:26.703 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2024-01-21T19:57:26.714 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:57:26.725 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:57:26.725 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:57:26.888 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:57:26.891 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:57:27.197 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:57:27.253 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:27.254 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:57:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:57:27.254 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-21T19:57:27.254 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-01-21T19:57:27.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.0 M 2024-01-21T19:57:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 15 M 2024-01-21T19:57:27.256 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:57:27.400 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T19:57:27.613 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 528 kB/s | 24 kB 00:00 2024-01-21T19:57:27.629 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T19:57:27.630 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 143 kB/s | 8.9 kB 00:00 2024-01-21T19:57:27.697 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2024-01-21T19:57:27.772 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2024-01-21T19:57:27.806 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 221 kB/s | 43 kB 00:00 2024-01-21T19:57:27.821 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T19:57:27.873 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-01-21T19:57:27.874 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:27.875 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.8 MB/s | 3.0 MB 00:00 2024-01-21T19:57:27.884 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:57:27.894 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:57:27.894 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:57:27.942 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T19:57:28.050 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:57:28.054 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:28.343 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:28.577 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-21T19:57:28.831 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-21T19:57:28.919 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T19:57:29.048 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-21T19:57:29.148 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:29.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-21T19:57:29.414 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:29.414 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:57:29.415 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T19:57:29.415 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T19:57:29.415 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T19:57:29.415 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T19:57:29.793 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T19:57:29.793 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:29.793 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:29.793 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T19:57:29.793 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T19:57:29.793 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T19:57:29.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T19:57:29.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T19:57:29.794 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T19:57:29.794 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:29.794 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:29.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T19:57:30.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-21T19:57:30.424 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:30.641 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:30 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:30.717 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:30.717 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:57:30.718 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-21T19:57:30.718 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-21T19:57:30.718 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-21T19:57:30.718 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-21T19:57:31.121 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:31.122 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:31.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T19:57:31.974 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:32.131 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T19:57:32.132 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout:Install 8 Packages 2024-01-21T19:57:32.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:32.134 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 37 M 2024-01-21T19:57:32.134 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 210 M 2024-01-21T19:57:32.134 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:57:32.539 INFO:teuthology.orchestra.run.smithi119.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 51 MB/s | 8.6 MB 00:00 2024-01-21T19:57:32.897 INFO:teuthology.orchestra.run.smithi119.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.0 MB/s | 4.7 MB 00:00 2024-01-21T19:57:32.964 INFO:teuthology.orchestra.run.smithi119.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.2 MB/s | 4.9 MB 00:00 2024-01-21T19:57:32.989 INFO:teuthology.orchestra.run.smithi119.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-01-21T19:57:33.090 INFO:teuthology.orchestra.run.smithi119.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.8 MB/s | 3.7 MB 00:00 2024-01-21T19:57:33.273 INFO:teuthology.orchestra.run.smithi119.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-01-21T19:57:33.316 INFO:teuthology.orchestra.run.smithi119.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 765 kB/s | 172 kB 00:00 2024-01-21T19:57:33.454 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-21T19:57:33.456 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout:Install 8 Packages 2024-01-21T19:57:33.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:33.458 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-01-21T19:57:33.458 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 210 M 2024-01-21T19:57:33.458 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:57:33.666 INFO:teuthology.orchestra.run.smithi119.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 20 MB/s | 14 MB 00:00 2024-01-21T19:57:33.666 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:33.667 INFO:teuthology.orchestra.run.smithi119.stdout:Total 24 MB/s | 37 MB 00:01 2024-01-21T19:57:33.896 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:57:33.914 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:57:33.914 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:57:33.985 INFO:teuthology.orchestra.run.smithi153.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 56 MB/s | 8.6 MB 00:00 2024-01-21T19:57:34.232 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:57:34.235 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:57:34.352 INFO:teuthology.orchestra.run.smithi153.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.3 MB/s | 4.9 MB 00:00 2024-01-21T19:57:34.402 INFO:teuthology.orchestra.run.smithi153.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-01-21T19:57:34.594 INFO:teuthology.orchestra.run.smithi153.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 6.2 MB/s | 4.7 MB 00:00 2024-01-21T19:57:34.661 INFO:teuthology.orchestra.run.smithi153.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.5 MB/s | 3.7 MB 00:00 2024-01-21T19:57:34.833 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:57:34.878 INFO:teuthology.orchestra.run.smithi153.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 796 kB/s | 172 kB 00:00 2024-01-21T19:57:34.953 INFO:teuthology.orchestra.run.smithi153.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-01-21T19:57:34.968 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:57:35.092 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:57:35.227 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:57:35.296 INFO:teuthology.orchestra.run.smithi153.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 16 MB/s | 14 MB 00:00 2024-01-21T19:57:35.300 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:35.303 INFO:teuthology.orchestra.run.smithi153.stdout:Total 20 MB/s | 37 MB 00:01 2024-01-21T19:57:35.536 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:57:35.553 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:57:35.553 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:57:35.822 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:57:35.884 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:57:35.887 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:35.993 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:57:36.520 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:36.605 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:57:36.645 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:57:36.760 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:57:36.778 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-21T19:57:36.945 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:57:37.435 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:57:37.526 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-21T19:57:37.695 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:57:38.023 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:57:38.268 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:57:38.297 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:57:38.453 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:57:39.138 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-21T19:57:39.490 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:57:39.770 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:57:40.026 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:57:41.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T19:57:41.521 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:57:41.533 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-21T19:57:41.834 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T19:57:41.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T19:57:42.235 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T19:57:42.235 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:42.235 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:42.235 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T19:57:42.235 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:42.236 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:42.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-01-21T19:57:42.966 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-21T19:57:43.059 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:43.176 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-21T19:57:43.483 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-21T19:57:43.484 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-21T19:57:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-21T19:57:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:43.919 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-21T19:57:43.919 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:43.920 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:44.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-01-21T19:57:44.552 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:44.553 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:57:44.553 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T19:57:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:44.555 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 1.9 M 2024-01-21T19:57:44.556 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 24 M 2024-01-21T19:57:44.556 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:57:44.755 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:44.856 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 621 kB/s | 27 kB 00:00 2024-01-21T19:57:44.940 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 157 kB/s | 20 kB 00:00 2024-01-21T19:57:44.957 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 185 kB/s | 27 kB 00:00 2024-01-21T19:57:45.032 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-01-21T19:57:45.057 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 630 kB/s | 126 kB 00:00 2024-01-21T19:57:45.175 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 713 kB/s | 155 kB 00:00 2024-01-21T19:57:45.191 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 268 kB/s | 43 kB 00:00 2024-01-21T19:57:45.258 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 164 kB/s | 33 kB 00:00 2024-01-21T19:57:45.350 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 93 kB/s | 16 kB 00:00 2024-01-21T19:57:45.375 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 442 kB/s | 81 kB 00:00 2024-01-21T19:57:45.425 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.2 MB/s | 61 kB 00:00 2024-01-21T19:57:45.451 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 640 kB/s | 64 kB 00:00 2024-01-21T19:57:45.526 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.4 MB/s | 1.2 MB 00:00 2024-01-21T19:57:45.528 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:45.528 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-01-21T19:57:45.580 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:57:45.595 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:57:45.595 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:57:45.681 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:57:45.684 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:57:46.171 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:57:46.224 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:46.226 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:57:46.226 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:46.226 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:46.227 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:57:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-01-21T19:57:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.9 M 2024-01-21T19:57:46.228 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 24 M 2024-01-21T19:57:46.229 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:57:46.403 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 718 kB/s | 27 kB 00:00 2024-01-21T19:57:46.411 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T19:57:46.495 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 205 kB/s | 27 kB 00:00 2024-01-21T19:57:46.520 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 129 kB/s | 20 kB 00:00 2024-01-21T19:57:46.570 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 755 kB/s | 126 kB 00:00 2024-01-21T19:57:46.595 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.1 MB/s | 110 kB 00:00 2024-01-21T19:57:46.606 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T19:57:46.729 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 269 kB/s | 43 kB 00:00 2024-01-21T19:57:46.754 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 208 kB/s | 33 kB 00:00 2024-01-21T19:57:46.780 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 597 kB/s | 155 kB 00:00 2024-01-21T19:57:46.843 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T19:57:46.880 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 648 kB/s | 81 kB 00:00 2024-01-21T19:57:46.906 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 93 kB/s | 16 kB 00:00 2024-01-21T19:57:46.989 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 592 kB/s | 64 kB 00:00 2024-01-21T19:57:47.048 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 431 kB/s | 61 kB 00:00 2024-01-21T19:57:47.065 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T19:57:47.165 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.0 MB/s | 1.2 MB 00:00 2024-01-21T19:57:47.165 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:47.166 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-01-21T19:57:47.223 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:57:47.239 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:57:47.239 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:57:47.285 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T19:57:47.327 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:57:47.330 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:47.511 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T19:57:47.772 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T19:57:47.834 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:48.054 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-21T19:57:48.242 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-21T19:57:48.354 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T19:57:48.479 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-21T19:57:48.733 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-21T19:57:48.967 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-21T19:57:49.191 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-21T19:57:49.384 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T19:57:49.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-21T19:57:49.664 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T19:57:49.724 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-21T19:57:49.865 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T19:57:50.065 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T19:57:50.233 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:57:50.515 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T19:57:50.516 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T19:57:50.754 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-21T19:57:50.957 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T19:57:50.958 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T19:57:50.959 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T19:57:50.959 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T19:57:50.959 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:50.959 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:51.016 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-21T19:57:51.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T19:57:51.258 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-21T19:57:51.434 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-21T19:57:51.609 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:57:51.797 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:51 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-21T19:57:51.914 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-21T19:57:51.915 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-21T19:57:52.442 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-21T19:57:52.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:52.442 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:52.442 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:52.443 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:52.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T19:57:53.282 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T19:57:53.283 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:53.284 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 5.8 M 2024-01-21T19:57:53.285 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 24 M 2024-01-21T19:57:53.285 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:57:53.302 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:53 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:53.701 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2024-01-21T19:57:53.859 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 134 kB/s | 21 kB 00:00 2024-01-21T19:57:53.926 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2024-01-21T19:57:54.026 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2024-01-21T19:57:54.143 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 225 kB/s | 49 kB 00:00 2024-01-21T19:57:54.543 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.4 MB/s | 4.8 MB 00:00 2024-01-21T19:57:54.545 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:54.545 INFO:teuthology.orchestra.run.smithi119.stdout:Total 4.6 MB/s | 5.8 MB 00:01 2024-01-21T19:57:54.595 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:57:54.604 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:57:54.604 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:57:54.704 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:57:54.707 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:57:54.794 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-21T19:57:54.796 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 5.8 M 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 24 M 2024-01-21T19:57:54.797 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:57:54.985 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:57:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-01-21T19:57:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T19:57:55.380 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 116 kB/s | 21 kB 00:00 2024-01-21T19:57:55.506 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2024-01-21T19:57:55.517 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T19:57:55.673 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.2 MB/s | 355 kB 00:00 2024-01-21T19:57:55.783 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 178 kB/s | 49 kB 00:00 2024-01-21T19:57:56.090 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.1 MB/s | 4.8 MB 00:00 2024-01-21T19:57:56.092 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:57:56.092 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.5 MB/s | 5.8 MB 00:01 2024-01-21T19:57:56.145 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:57:56.155 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:57:56.155 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:57:56.259 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:57:56.263 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:57:56.422 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T19:57:56.531 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:57:56.767 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T19:57:56.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-21T19:57:56.977 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T19:57:57.144 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-21T19:57:57.181 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:57:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T19:57:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T19:57:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T19:57:57.466 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T19:57:57.846 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:57:57.847 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:57:57.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-01-21T19:57:58.064 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-21T19:57:58.396 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-21T19:57:58.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-21T19:57:58.687 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:58 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:57:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:59.121 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-21T19:57:59.121 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-21T19:57:59.121 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-21T19:57:59.121 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-21T19:57:59.121 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-21T19:57:59.122 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:57:59.550 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:57:59.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-01-21T19:58:00.167 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:00.167 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-01-21T19:58:00.168 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:00.169 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 968 k 2024-01-21T19:58:00.169 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 3.2 M 2024-01-21T19:58:00.169 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:58:00.261 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-01-21T19:58:00.412 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 409 kB/s | 83 kB 00:00 2024-01-21T19:58:00.412 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:00.412 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.9 MB/s | 968 kB 00:00 2024-01-21T19:58:00.422 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:58:00.423 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:00 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:00.442 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:58:00.442 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:58:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:58:00.531 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:58:00.696 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:58:00.997 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T19:58:01.181 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:58:01.570 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:58:01.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T19:58:01.917 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-21T19:58:01.918 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 968 k 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 3.2 M 2024-01-21T19:58:01.919 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:58:01.971 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T19:58:01.972 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:01.972 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:58:01.972 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T19:58:01.972 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:01.972 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:02.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-01-21T19:58:02.922 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:03.592 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-01-21T19:58:03.750 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 394 kB/s | 83 kB 00:00 2024-01-21T19:58:03.751 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:03.751 INFO:teuthology.orchestra.run.smithi153.stdout:Total 528 kB/s | 968 kB 00:01 2024-01-21T19:58:03.763 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:58:03.785 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:58:03.785 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:58:03.878 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:58:03.881 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:58:04.056 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:58:04.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T19:58:04.405 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 131 k 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 439 k 2024-01-21T19:58:04.406 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:58:04.452 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 131 kB 00:00 2024-01-21T19:58:04.452 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:04.453 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.7 MB/s | 131 kB 00:00 2024-01-21T19:58:04.453 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:58:04.458 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:58:04.458 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:58:04.507 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:58:04.511 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:58:04.524 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:58:04.625 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:58:04.838 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:04.935 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-21T19:58:04.935 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-21T19:58:05.130 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:05.348 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-21T19:58:05.348 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:05.348 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:58:05.348 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-21T19:58:05.348 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:05.348 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:05.493 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:05.494 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:05.494 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:58:05.494 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:05.494 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:05.494 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:05.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-01-21T19:58:05.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-01-21T19:58:06.346 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:06 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:06.433 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:06 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:07.831 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:07.832 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:07.832 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:07.832 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 131 k 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 439 k 2024-01-21T19:58:07.833 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:58:07.865 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:07.911 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:07.912 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:07.912 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:08.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-01-21T19:58:08.378 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 241 kB/s | 131 kB 00:00 2024-01-21T19:58:08.378 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:08.379 INFO:teuthology.orchestra.run.smithi153.stdout:Total 240 kB/s | 131 kB 00:00 2024-01-21T19:58:08.379 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:58:08.386 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:58:08.386 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:58:08.442 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:58:08.445 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:58:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:58:08.734 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:08 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:08.772 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:09.080 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:09.440 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:09.440 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:09.440 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:58:09.440 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:09.440 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:09.440 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:09.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-01-21T19:58:10.226 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:10.227 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:58:10.228 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:10.228 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:58:10.228 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:10.228 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 24 k 2024-01-21T19:58:10.228 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 94 k 2024-01-21T19:58:10.228 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:58:10.282 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 442 kB/s | 24 kB 00:00 2024-01-21T19:58:10.283 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:10.283 INFO:teuthology.orchestra.run.smithi119.stdout:Total 432 kB/s | 24 kB 00:00 2024-01-21T19:58:10.283 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:58:10.287 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:58:10.287 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:58:10.289 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:58:10.292 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:58:10.342 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:58:10.372 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:10 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:10.528 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:10.798 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:11.136 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:11.136 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:11.136 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:58:11.136 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:11.136 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:11.136 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:11.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-01-21T19:58:11.796 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:11.840 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:11.842 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:11.842 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:11.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-01-21T19:58:11.983 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:11 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:12.680 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:12 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:13.423 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:13.468 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:13.469 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:13.470 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:13.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-21T19:58:14.165 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 24 k 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 94 k 2024-01-21T19:58:14.166 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:58:14.301 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:14 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:14.309 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 171 kB/s | 24 kB 00:00 2024-01-21T19:58:14.310 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:14.310 INFO:teuthology.orchestra.run.smithi153.stdout:Total 169 kB/s | 24 kB 00:00 2024-01-21T19:58:14.310 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:58:14.313 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:58:14.313 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:58:14.315 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:58:14.319 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:58:14.369 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:58:14.571 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:14.832 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:15.214 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:58:15.214 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:15.215 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:15.215 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:15.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-01-21T19:58:15.731 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:15.777 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:15.778 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:15.778 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:15.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-01-21T19:58:16.058 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:16 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:16.625 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:16 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:17.511 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:17.556 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:17.558 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:17.558 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:17.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-01-21T19:58:18.071 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:18.117 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:18.118 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:18.118 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:18.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-01-21T19:58:18.389 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:18.948 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:18 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:19.839 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:19.884 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:19.886 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:19.887 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:19.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-01-21T19:58:20.386 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:20.432 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:20.433 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:20.433 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:20.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-01-21T19:58:20.731 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:20 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:21.259 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:21 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:22.167 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:22.212 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:22.213 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:22.213 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:22.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-01-21T19:58:22.702 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:22.746 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:22.748 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:22.748 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:22.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-01-21T19:58:23.054 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:23.584 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:24.504 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:24.549 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:24.550 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:24.551 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:24.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-01-21T19:58:25.015 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:25.060 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:25.061 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-01-21T19:58:25.061 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:25.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-01-21T19:58:25.427 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:25.884 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:25 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:26.890 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:26.935 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:26.936 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:26.936 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:27.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-01-21T19:58:27.363 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:27.363 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:58:27.364 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:27.365 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 89 k 2024-01-21T19:58:27.365 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 298 k 2024-01-21T19:58:27.365 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:58:27.405 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 89 kB 00:00 2024-01-21T19:58:27.405 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:27.406 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.1 MB/s | 89 kB 00:00 2024-01-21T19:58:27.406 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:58:27.415 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:58:27.416 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:58:27.465 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:58:27.468 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:58:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:58:27.774 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:27 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:27.792 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:28.485 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:28.486 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:28.486 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:58:28.486 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:28.486 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:28.486 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:28.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-01-21T19:58:29.224 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-21T19:58:29.270 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:29.271 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-01-21T19:58:29.271 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:29.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-01-21T19:58:29.446 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:29 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:30.113 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:30 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:30.905 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:58:30.906 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:30.907 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.7 M 2024-01-21T19:58:30.907 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 18 M 2024-01-21T19:58:30.907 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:58:31.000 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 40 MB/s | 3.7 MB 00:00 2024-01-21T19:58:31.000 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:31.000 INFO:teuthology.orchestra.run.smithi119.stdout:Total 40 MB/s | 3.7 MB 00:00 2024-01-21T19:58:31.001 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:58:31.012 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:58:31.012 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:58:31.075 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:58:31.078 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:58:31.542 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:58:31.594 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:31.595 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:58:31.596 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:31.596 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 89 k 2024-01-21T19:58:31.596 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 298 k 2024-01-21T19:58:31.596 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:58:31.661 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.3 MB/s | 89 kB 00:00 2024-01-21T19:58:31.662 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:31.662 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 89 kB 00:00 2024-01-21T19:58:31.662 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:58:31.672 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:58:31.672 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:58:31.720 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:58:31.723 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:58:31.726 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:31.827 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:58:32.035 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:32.129 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:32.326 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:32.489 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:32.489 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:32.489 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:58:32.489 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:32.489 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:32.489 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:32.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-01-21T19:58:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:32.750 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:58:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:32.750 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:32.750 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:32.947 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-01-21T19:58:33.423 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:02:33 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:33.685 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:33 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:34.906 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-01-21T19:58:34.907 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:34.908 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 173 k 2024-01-21T19:58:34.908 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 649 k 2024-01-21T19:58:34.908 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-01-21T19:58:34.954 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2024-01-21T19:58:34.954 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:34.954 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.6 MB/s | 173 kB 00:00 2024-01-21T19:58:34.954 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-01-21T19:58:34.965 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-01-21T19:58:34.965 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-01-21T19:58:35.015 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-01-21T19:58:35.018 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-01-21T19:58:35.132 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-01-21T19:58:35.177 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:35.177 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:35.177 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:35.178 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.7 M 2024-01-21T19:58:35.179 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 18 M 2024-01-21T19:58:35.179 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:58:35.260 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2024-01-21T19:58:35.261 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:35.261 INFO:teuthology.orchestra.run.smithi153.stdout:Total 45 MB/s | 3.7 MB 00:00 2024-01-21T19:58:35.262 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:58:35.273 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:58:35.273 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:58:35.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:35.337 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:58:35.340 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:58:35.618 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:35.829 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:58:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-01-21T19:58:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-01-21T19:58:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-01-21T19:58:36.027 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:36.248 DEBUG:teuthology.parallel:result is None 2024-01-21T19:58:36.426 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:36.786 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:36.786 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:36.786 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:58:36.786 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:36.786 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:36.786 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:36.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-01-21T19:58:37.713 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:37 ago on Sun 21 Jan 2024 07:56:00 PM UTC. 2024-01-21T19:58:39.195 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-01-21T19:58:39.196 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:39.197 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 173 k 2024-01-21T19:58:39.197 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 649 k 2024-01-21T19:58:39.197 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-01-21T19:58:39.247 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 173 kB 00:00 2024-01-21T19:58:39.247 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-01-21T19:58:39.247 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.4 MB/s | 173 kB 00:00 2024-01-21T19:58:39.247 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-01-21T19:58:39.258 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-01-21T19:58:39.258 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-01-21T19:58:39.308 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-01-21T19:58:39.311 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-01-21T19:58:39.428 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-01-21T19:58:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:39.945 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:40.306 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-21T19:58:40.306 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:40.306 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-01-21T19:58:40.307 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-21T19:58:40.307 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T19:58:40.307 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-01-21T19:58:40.544 DEBUG:teuthology.parallel:result is None 2024-01-21T19:58:40.544 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:58:40.544 INFO:teuthology.packaging:ref: None 2024-01-21T19:58:40.544 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:58:40.544 INFO:teuthology.packaging:branch: None 2024-01-21T19:58:40.544 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:58:40.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:58:40.688 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T19:58:40.716 INFO:teuthology.orchestra.run.smithi119.stdout:16.2.4-0.el8 2024-01-21T19:58:40.716 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T19:58:40.716 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T19:58:40.718 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-21T19:58:40.718 INFO:teuthology.packaging:ref: None 2024-01-21T19:58:40.718 INFO:teuthology.packaging:tag: v16.2.4 2024-01-21T19:58:40.718 INFO:teuthology.packaging:branch: None 2024-01-21T19:58:40.718 INFO:teuthology.packaging:sha1: ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec 2024-01-21T19:58:40.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-21T19:58:40.859 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T19:58:40.881 INFO:teuthology.orchestra.run.smithi153.stdout:16.2.4-0.el8 2024-01-21T19:58:40.882 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-21T19:58:40.882 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-21T19:58:40.883 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T19:58:40.884 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:58:40.884 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T19:58:40.917 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T19:58:40.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T19:58:40.948 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T19:58:40.948 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:58:40.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T19:58:40.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T19:58:41.056 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T19:58:41.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T19:58:41.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T19:58:41.148 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T19:58:41.149 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:58:41.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T19:58:41.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T19:58:41.246 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T19:58:41.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T19:58:41.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T19:58:41.340 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T19:58:41.340 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:58:41.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T19:58:41.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T19:58:41.436 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T19:58:41.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T19:58:41.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T19:58:41.530 INFO:teuthology.run_tasks:Running task print... 2024-01-21T19:58:41.540 INFO:teuthology.task.print:**** done install task... 2024-01-21T19:58:41.540 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-21T19:58:41.636 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec'} 2024-01-21T19:58:41.636 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-21T19:58:41.637 INFO:tasks.cephadm:Cluster fsid is 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:41.637 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-21T19:58:41.637 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-21T19:58:41.637 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi119': '172.21.15.119', 'mon.smithi153': '172.21.15.153'} 2024-01-21T19:58:41.637 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-21T19:58:41.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-01-21T19:58:41.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-01-21T19:58:41.694 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-21T19:58:41.695 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:41.843 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 19:58 /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:41.843 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:41.974 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 21 19:58 /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:41.975 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:41.996 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-21T19:58:42.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-01-21T19:58:42.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-01-21T19:58:42.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-01-21T19:58:42.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-01-21T19:58:42.161 INFO:tasks.cephadm:Writing seed config... 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-21T19:58:42.162 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-21T19:58:42.163 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-21T19:58:42.164 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-21T19:58:42.165 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T19:58:42.165 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-21T19:58:42.194 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 79b26e34-b897-11ee-95b0-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-21T19:58:42.195 DEBUG:teuthology.orchestra.run.smithi119:mon.smithi119> sudo journalctl -f -n 0 -u ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119.service 2024-01-21T19:58:42.237 INFO:tasks.cephadm:Bootstrapping... 2024-01-21T19:58:42.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 79b26e34-b897-11ee-95b0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.119 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:58:42.264 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:-- Logs begin at Sun 2024-01-21 19:49:13 UTC. -- 2024-01-21T19:58:42.496 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:58:43.309 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: 4.6.1 2024-01-21T19:58:43.316 INFO:teuthology.orchestra.run.smithi119.stderr:Verifying podman|docker is present... 2024-01-21T19:58:43.316 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:58:43.338 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: 4.6.1 2024-01-21T19:58:43.347 INFO:teuthology.orchestra.run.smithi119.stderr:Verifying lvm2 is present... 2024-01-21T19:58:43.347 INFO:teuthology.orchestra.run.smithi119.stderr:Verifying time synchronization is in place... 2024-01-21T19:58:43.347 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T19:58:43.354 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T19:58:43.355 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active chrony.service 2024-01-21T19:58:43.361 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: inactive 2024-01-21T19:58:43.362 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T19:58:43.367 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: enabled 2024-01-21T19:58:43.368 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T19:58:43.373 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: active 2024-01-21T19:58:43.374 INFO:teuthology.orchestra.run.smithi119.stderr:Unit chronyd.service is enabled and running 2024-01-21T19:58:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:Repeating the final host check... 2024-01-21T19:58:43.375 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-21T19:58:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: 4.6.1 2024-01-21T19:58:43.405 INFO:teuthology.orchestra.run.smithi119.stderr:podman|docker (/bin/podman) is present 2024-01-21T19:58:43.406 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl is present 2024-01-21T19:58:43.406 INFO:teuthology.orchestra.run.smithi119.stderr:lvcreate is present 2024-01-21T19:58:43.406 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled chrony.service 2024-01-21T19:58:43.411 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-21T19:58:43.412 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active chrony.service 2024-01-21T19:58:43.417 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: inactive 2024-01-21T19:58:43.417 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-21T19:58:43.423 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: enabled 2024-01-21T19:58:43.423 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active chronyd.service 2024-01-21T19:58:43.428 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: active 2024-01-21T19:58:43.429 INFO:teuthology.orchestra.run.smithi119.stderr:Unit chronyd.service is enabled and running 2024-01-21T19:58:43.429 INFO:teuthology.orchestra.run.smithi119.stderr:Host looks OK 2024-01-21T19:58:43.429 INFO:teuthology.orchestra.run.smithi119.stderr:Cluster fsid: 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:58:43.429 INFO:teuthology.orchestra.run.smithi119.stderr:Acquiring lock 140176363207144 on /run/cephadm/79b26e34-b897-11ee-95b0-87774f69a715.lock 2024-01-21T19:58:43.430 INFO:teuthology.orchestra.run.smithi119.stderr:Lock 140176363207144 acquired on /run/cephadm/79b26e34-b897-11ee-95b0-87774f69a715.lock 2024-01-21T19:58:43.430 INFO:teuthology.orchestra.run.smithi119.stderr:Verifying IP 172.21.15.119 port 3300 ... 2024-01-21T19:58:43.430 INFO:teuthology.orchestra.run.smithi119.stderr:Verifying IP 172.21.15.119 port 6789 ... 2024-01-21T19:58:43.431 INFO:teuthology.orchestra.run.smithi119.stderr:Base mon IP is 172.21.15.119, final addrv is [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-01-21T19:58:43.431 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /sbin/ip route ls 2024-01-21T19:58:43.433 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.119 metric 100 2024-01-21T19:58:43.433 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.119 metric 100 2024-01-21T19:58:43.433 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /sbin/ip -6 route ls 2024-01-21T19:58:43.436 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-21T19:58:43.436 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-21T19:58:43.436 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-01-21T19:58:43.436 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /sbin/ip -6 addr ls 2024-01-21T19:58:43.438 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-21T19:58:43.438 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-21T19:58:43.439 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T19:58:43.439 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-21T19:58:43.439 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-01-21T19:58:43.439 INFO:teuthology.orchestra.run.smithi119.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-21T19:58:43.440 INFO:teuthology.orchestra.run.smithi119.stderr:Mon IP 172.21.15.119 is in CIDR network 172.21.0.0/20 2024-01-21T19:58:43.440 INFO:teuthology.orchestra.run.smithi119.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-21T19:58:43.441 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-21T19:58:43.441 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-21T19:58:43.530 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-21T19:58:50.011 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: Getting image source signatures 2024-01-21T19:58:50.147 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-21T19:58:50.147 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-21T19:59:01.976 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T19:59:02.034 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: Writing manifest to image destination 2024-01-21T19:59:02.353 INFO:teuthology.orchestra.run.smithi119.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-21T19:59:02.361 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-21T19:59:02.898 INFO:teuthology.orchestra.run.smithi119.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T19:59:03.262 INFO:teuthology.orchestra.run.smithi119.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-21T19:59:03.263 INFO:teuthology.orchestra.run.smithi119.stderr:Extracting ceph user uid/gid from container image... 2024-01-21T19:59:03.263 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-21T19:59:03.565 INFO:teuthology.orchestra.run.smithi119.stderr:stat: 167 167 2024-01-21T19:59:03.879 INFO:teuthology.orchestra.run.smithi119.stderr:Creating initial keys... 2024-01-21T19:59:03.879 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:59:04.171 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-authtool: AQCId61lUADnCRAA/TAczAAgqITLLQY0EzA5sg== 2024-01-21T19:59:04.512 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:59:04.813 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-authtool: AQCId61lp/wsMBAAszwkrrccWVBRjZibC3jIRA== 2024-01-21T19:59:05.149 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-21T19:59:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-authtool: AQCJd61lmEwZGRAAE0X9Qd2L+b6XluJaO7QBRA== 2024-01-21T19:59:05.762 INFO:teuthology.orchestra.run.smithi119.stderr:Creating initial monmap... 2024-01-21T19:59:05.763 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpy10efdc5:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 79b26e34-b897-11ee-95b0-87774f69a715 --addv smithi119 [v2:172.21.15.119:3300,v1:172.21.15.119:6789] /tmp/monmap 2024-01-21T19:59:06.047 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T19:59:06.048 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:06.048 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T19:59:06.378 INFO:teuthology.orchestra.run.smithi119.stderr:monmaptool for smithi119 [v2:172.21.15.119:3300,v1:172.21.15.119:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-21T19:59:06.379 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/monmaptool: set fsid to 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:06.379 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-21T19:59:06.379 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T19:59:06.379 INFO:teuthology.orchestra.run.smithi119.stderr:Creating mon... 2024-01-21T19:59:06.379 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/mon.smithi119:/var/lib/ceph/mon/ceph-smithi119:z -v /tmp/ceph-tmpo9rn8yaj:/tmp/keyring:z -v /tmp/ceph-tmpy10efdc5:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi119 --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.661+0000 7f8932a62700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.661+0000 7f8932a62700 1 imported monmap: 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: fsid 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: last_changed 2024-01-21T19:59:06.037046+0000 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: created 2024-01-21T19:59:06.037046+0000 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.smithi119 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.661+0000 7f8932a62700 0 /usr/bin/ceph-mon: set fsid to 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Compile date May 13 2021 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: DB SUMMARY 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi119/store.db dir, Total Num: 0, files: 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi119/store.db: 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.create_if_missing: 1 2024-01-21T19:59:06.682 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.env: 0x56139ebc11c0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.fs: Posix File System 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.info_log: 0x5613a00a4ec0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.statistics: (nil) 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.use_fsync: 0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T19:59:06.683 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.db_log_dir: 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi119/store.db 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T19:59:06.684 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.write_buffer_manager: 0x5613a00ab530 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.unordered_write: 0 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T19:59:06.685 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.row_cache: None 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.wal_filter: None 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T19:59:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T19:59:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.max_open_files: -1 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Compression algorithms supported: 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T19:59:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kZSTD supported: 0 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.662+0000 7f8932a62700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi119/store.db/MANIFEST-000001 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T19:59:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.merge_operator: 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_filter: None 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56139ffbc120) 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-21T19:59:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_cache: 0x56139fff2d10 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-21T19:59:06.691 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression: NoCompression 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T19:59:06.692 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.num_levels: 7 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T19:59:06.693 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T19:59:06.694 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T19:59:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T19:59:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.ttl: 2592000 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi119/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.663+0000 7f8932a62700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.664+0000 7f8932a62700 4 rocksdb: DB pointer 0x5613a00b7800 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.664+0000 7f891b8fb700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.664+0000 7f891b8fb700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T19:59:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:59:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:59:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: 2024-01-21T19:59:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.666+0000 7f8932a62700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-21T19:59:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.666+0000 7f8932a62700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-21T19:59:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph-mon: debug 2024-01-21T19:59:06.666+0000 7f8932a62700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi119 for mon.smithi119 2024-01-21T19:59:07.003 INFO:teuthology.orchestra.run.smithi119.stderr:create mon.smithi119 on 2024-01-21T19:59:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:07.008 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl disable ceph.target 2024-01-21T19:59:07.015 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-21T19:59:07.098 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl enable ceph.target 2024-01-21T19:59:07.105 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-21T19:59:07.184 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl start ceph.target 2024-01-21T19:59:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl enable ceph-79b26e34-b897-11ee-95b0-87774f69a715.target 2024-01-21T19:59:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-79b26e34-b897-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-79b26e34-b897-11ee-95b0-87774f69a715.target. 2024-01-21T19:59:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-79b26e34-b897-11ee-95b0-87774f69a715.target → /etc/systemd/system/ceph-79b26e34-b897-11ee-95b0-87774f69a715.target. 2024-01-21T19:59:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715.target 2024-01-21T19:59:07.281 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl daemon-reload 2024-01-21T19:59:07.363 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119 2024-01-21T19:59:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl reset-failed ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119 2024-01-21T19:59:07.398 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Failed to reset failed state of unit ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119.service: Unit ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119.service not loaded. 2024-01-21T19:59:07.398 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl enable ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119 2024-01-21T19:59:07.405 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Created symlink /etc/systemd/system/ceph-79b26e34-b897-11ee-95b0-87774f69a715.target.wants/ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119.service → /etc/systemd/system/ceph-79b26e34-b897-11ee-95b0-87774f69a715@.service. 2024-01-21T19:59:07.491 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119 2024-01-21T19:59:07.824 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:59:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: disabled 2024-01-21T19:59:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:59:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: inactive 2024-01-21T19:59:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:firewalld.service is not enabled 2024-01-21T19:59:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T19:59:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T19:59:07.842 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for mon to start... 2024-01-21T19:59:07.842 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for mon... 2024-01-21T19:59:07.842 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/mon.smithi119:/var/lib/ceph/mon/ceph-smithi119:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-21T19:59:08.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: cluster 2024-01-21T19:59:07.827555+0000 mon.smithi119 (mon.0) 1 : cluster [INF] mon.smithi119 is new leader, mons smithi119 in quorum (ranks 0) 2024-01-21T19:59:08.225 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.233+0000 7f831b7f0700 1 Processor -- start 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.234+0000 7f831b7f0700 1 -- start start 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.234+0000 7f831b7f0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 0x7f83140fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.234+0000 7f831b7f0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8314070790 con 0x7f8314070370 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.234+0000 7f831958c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 0x7f83140fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.234+0000 7f831958c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 0x7f83140fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42118/0 (socket says 172.21.15.119:42118) 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.234+0000 7f831958c700 1 -- 172.21.15.119:0/3696888214 learned_addr learned my addr 172.21.15.119:0/3696888214 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831958c700 1 -- 172.21.15.119:0/3696888214 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83140fee90 con 0x7f8314070370 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831958c700 1 --2- 172.21.15.119:0/3696888214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 0x7f83140fe9a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f830400b5f0 tx=0x7f8304006ee0).ready entity=mon.0 client_cookie=9044086283f7337f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f8303fff700 1 -- 172.21.15.119:0/3696888214 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f830400d600 con 0x7f8314070370 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f8303fff700 1 -- 172.21.15.119:0/3696888214 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830400d760 con 0x7f8314070370 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f8303fff700 1 -- 172.21.15.119:0/3696888214 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f830400da30 con 0x7f8314070370 2024-01-21T19:59:08.228 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831b7f0700 1 -- 172.21.15.119:0/3696888214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 msgr2=0x7f83140fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:08.229 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831b7f0700 1 --2- 172.21.15.119:0/3696888214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 0x7f83140fe9a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f830400b5f0 tx=0x7f8304006ee0).stop 2024-01-21T19:59:08.229 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831b7f0700 1 -- 172.21.15.119:0/3696888214 shutdown_connections 2024-01-21T19:59:08.229 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831b7f0700 1 --2- 172.21.15.119:0/3696888214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8314070370 0x7f83140fe9a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:08.229 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.235+0000 7f831b7f0700 1 -- 172.21.15.119:0/3696888214 >> 172.21.15.119:0/3696888214 conn(0x7f83140f63c0 msgr2=0x7f83140f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:08.229 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831b7f0700 1 -- 172.21.15.119:0/3696888214 shutdown_connections 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831b7f0700 1 -- 172.21.15.119:0/3696888214 wait complete. 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831b7f0700 1 Processor -- start 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831b7f0700 1 -- start start 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831b7f0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 0x7f83140fa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831b7f0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83140fc800 con 0x7f83140fc3e0 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831958c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 0x7f83140fa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831958c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 0x7f83140fa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42132/0 (socket says 172.21.15.119:42132) 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.236+0000 7f831958c700 1 -- 172.21.15.119:0/1555661034 learned_addr learned my addr 172.21.15.119:0/1555661034 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f831958c700 1 -- 172.21.15.119:0/1555661034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830400b200 con 0x7f83140fc3e0 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f831958c700 1 --2- 172.21.15.119:0/1555661034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 0x7f83140fa8a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8304006f20 tx=0x7f8304016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:08.231 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8304011b00 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8304011c80 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83140fad90 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f830401d430 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83140fb200 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f830400f070 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.237+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8304027460 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.238+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8314045ad0 con 0x7f83140fc3e0 2024-01-21T19:59:08.232 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.240+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8304016070 con 0x7f83140fc3e0 2024-01-21T19:59:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.306+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f83140ff1e0 con 0x7f83140fc3e0 2024-01-21T19:59:08.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.306+0000 7f8301ffb700 1 -- 172.21.15.119:0/1555661034 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f830401d5f0 con 0x7f83140fc3e0 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: cluster: 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: id: 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: services: 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi119 (age 0.465162s) 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: data: 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-21T19:59:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: pgs: 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 msgr2=0x7f83140fa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 --2- 172.21.15.119:0/1555661034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 0x7f83140fa8a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8304006f20 tx=0x7f8304016000).stop 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 shutdown_connections 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 --2- 172.21.15.119:0/1555661034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83140fc3e0 0x7f83140fa8a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 >> 172.21.15.119:0/1555661034 conn(0x7f83140f63c0 msgr2=0x7f83140f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 shutdown_connections 2024-01-21T19:59:08.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:08.307+0000 7f831b7f0700 1 -- 172.21.15.119:0/1555661034 wait complete. 2024-01-21T19:59:08.629 INFO:teuthology.orchestra.run.smithi119.stderr:mon is available 2024-01-21T19:59:08.629 INFO:teuthology.orchestra.run.smithi119.stderr:Assimilating anything we can from ceph.conf... 2024-01-21T19:59:08.629 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z -v /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/mon.smithi119:/var/lib/ceph/mon/ceph-smithi119:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi119/config 2024-01-21T19:59:09.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.031+0000 7f8b9ae1b700 1 Processor -- start 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.031+0000 7f8b9ae1b700 1 -- start start 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.031+0000 7f8b9ae1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b940fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.031+0000 7f8b9ae1b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b940fb490 con 0x7f8b940fab80 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b98bb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b940fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b98bb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b940fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42144/0 (socket says 172.21.15.119:42144) 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b98bb7700 1 -- 172.21.15.119:0/1953282019 learned_addr learned my addr 172.21.15.119:0/1953282019 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b98bb7700 1 -- 172.21.15.119:0/1953282019 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b940fb5d0 con 0x7f8b940fab80 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b98bb7700 1 --2- 172.21.15.119:0/1953282019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b940fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8b8800b870 tx=0x7f8b88006f70).ready entity=mon.0 client_cookie=a1cd5f635d2feec6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b937fe700 1 -- 172.21.15.119:0/1953282019 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8800da20 con 0x7f8b940fab80 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b937fe700 1 -- 172.21.15.119:0/1953282019 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8800db80 con 0x7f8b940fab80 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.032+0000 7f8b937fe700 1 -- 172.21.15.119:0/1953282019 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8800de50 con 0x7f8b940fab80 2024-01-21T19:59:09.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/1953282019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 msgr2=0x7f8b940fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 --2- 172.21.15.119:0/1953282019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b940fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8b8800b870 tx=0x7f8b88006f70).stop 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/1953282019 shutdown_connections 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 --2- 172.21.15.119:0/1953282019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b940fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/1953282019 >> 172.21.15.119:0/1953282019 conn(0x7f8b940f63c0 msgr2=0x7f8b940f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/1953282019 shutdown_connections 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.033+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/1953282019 wait complete. 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b9ae1b700 1 Processor -- start 2024-01-21T19:59:09.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b9ae1b700 1 -- start start 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b9ae1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b94108040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b9ae1b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94108530 con 0x7f8b940fab80 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b98bb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b94108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b98bb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b94108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42158/0 (socket says 172.21.15.119:42158) 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b98bb7700 1 -- 172.21.15.119:0/3706096152 learned_addr learned my addr 172.21.15.119:0/3706096152 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.034+0000 7f8b98bb7700 1 -- 172.21.15.119:0/3706096152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8800b480 con 0x7f8b940fab80 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b98bb7700 1 --2- 172.21.15.119:0/3706096152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b94108040 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8b88006c70 tx=0x7f8b88006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8801b110 con 0x7f8b940fab80 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8801b270 con 0x7f8b940fab80 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b94108730 con 0x7f8b940fab80 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b880105d0 con 0x7f8b940fab80 2024-01-21T19:59:09.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b94108b40 con 0x7f8b940fab80 2024-01-21T19:59:09.030 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8b8802c070 con 0x7f8b940fab80 2024-01-21T19:59:09.030 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.035+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8b880197c0 con 0x7f8b940fab80 2024-01-21T19:59:09.030 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.036+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b94045ad0 con 0x7f8b940fab80 2024-01-21T19:59:09.030 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.038+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8b880449c0 con 0x7f8b940fab80 2024-01-21T19:59:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: cluster 2024-01-21T19:59:07.843033+0000 mon.smithi119 (mon.0) 2 : cluster [INF] mon.smithi119 is new leader, mons smithi119 in quorum (ranks 0) 2024-01-21T19:59:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: cluster 2024-01-21T19:59:07.843281+0000 mon.smithi119 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi119=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} 2024-01-21T19:59:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: cluster 2024-01-21T19:59:07.846759+0000 mon.smithi119 (mon.0) 4 : cluster [DBG] fsmap 2024-01-21T19:59:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: cluster 2024-01-21T19:59:07.848744+0000 mon.smithi119 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T19:59:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: cluster 2024-01-21T19:59:07.849246+0000 mon.smithi119 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T19:59:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: audit 2024-01-21T19:59:08.308161+0000 mon.smithi119 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.119:0/1555661034' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-21T19:59:09.094 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.102+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8b940fbec0 con 0x7f8b940fab80 2024-01-21T19:59:09.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.105+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b88019e70 con 0x7f8b940fab80 2024-01-21T19:59:09.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.105+0000 7f8b917fa700 1 -- 172.21.15.119:0/3706096152 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f8b88033020 con 0x7f8b940fab80 2024-01-21T19:59:09.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: [global] 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: fsid = 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: [mgr] 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: [osd] 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-21T19:59:09.100 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 msgr2=0x7f8b94108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 --2- 172.21.15.119:0/3706096152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b94108040 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8b88006c70 tx=0x7f8b88006f70).stop 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 shutdown_connections 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 --2- 172.21.15.119:0/3706096152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b940fab80 0x7f8b94108040 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 >> 172.21.15.119:0/3706096152 conn(0x7f8b940f63c0 msgr2=0x7f8b940f85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 shutdown_connections 2024-01-21T19:59:09.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.106+0000 7f8b9ae1b700 1 -- 172.21.15.119:0/3706096152 wait complete. 2024-01-21T19:59:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:Generating new minimal ceph.conf... 2024-01-21T19:59:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z -v /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/mon.smithi119:/var/lib/ceph/mon/ceph-smithi119:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi119/config 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.797+0000 7f9febe1a700 1 Processor -- start 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.797+0000 7f9febe1a700 1 -- start start 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.797+0000 7f9febe1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.797+0000 7f9febe1a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40fedd0 con 0x7f9fe4070430 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe9bb6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe9bb6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42168/0 (socket says 172.21.15.119:42168) 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe9bb6700 1 -- 172.21.15.119:0/815146816 learned_addr learned my addr 172.21.15.119:0/815146816 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe9bb6700 1 -- 172.21.15.119:0/815146816 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe40fef10 con 0x7f9fe4070430 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe9bb6700 1 --2- 172.21.15.119:0/815146816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9fd400b5f0 tx=0x7f9fd4006ee0).ready entity=mon.0 client_cookie=752480b58a6bd55e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe8bb4700 1 -- 172.21.15.119:0/815146816 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd400d600 con 0x7f9fe4070430 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe8bb4700 1 -- 172.21.15.119:0/815146816 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9fd400dbd0 con 0x7f9fe4070430 2024-01-21T19:59:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.798+0000 7f9fe8bb4700 1 -- 172.21.15.119:0/815146816 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd4011c00 con 0x7f9fe4070430 2024-01-21T19:59:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- 172.21.15.119:0/815146816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 msgr2=0x7f9fe40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 --2- 172.21.15.119:0/815146816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9fd400b5f0 tx=0x7f9fd4006ee0).stop 2024-01-21T19:59:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- 172.21.15.119:0/815146816 shutdown_connections 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 --2- 172.21.15.119:0/815146816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- 172.21.15.119:0/815146816 >> 172.21.15.119:0/815146816 conn(0x7f9fe40f63c0 msgr2=0x7f9fe40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- 172.21.15.119:0/815146816 shutdown_connections 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- 172.21.15.119:0/815146816 wait complete. 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 Processor -- start 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- start start 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.799+0000 7f9febe1a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40fa7e0 con 0x7f9fe4070430 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fe9bb6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:09.794 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fe9bb6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42170/0 (socket says 172.21.15.119:42170) 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fe9bb6700 1 -- 172.21.15.119:0/49587774 learned_addr learned my addr 172.21.15.119:0/49587774 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fe9bb6700 1 -- 172.21.15.119:0/49587774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd400b200 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fe9bb6700 1 --2- 172.21.15.119:0/49587774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9fd400e040 tx=0x7f9fd4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd400d9f0 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9fd401a9f0 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe40fa980 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe40fad90 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd401b8c0 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9fd400f070 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.800+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9fd4025b30 con 0x7f9fe4070430 2024-01-21T19:59:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.801+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4045ad0 con 0x7f9fe4070430 2024-01-21T19:59:09.796 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.804+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9fd4016070 con 0x7f9fe4070430 2024-01-21T19:59:09.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.868+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9fe4104820 con 0x7f9fe4070430 2024-01-21T19:59:09.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9fd27fc700 1 -- 172.21.15.119:0/49587774 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f9fd401bd70 con 0x7f9fe4070430 2024-01-21T19:59:09.862 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 msgr2=0x7f9fe40fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:09.862 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 --2- 172.21.15.119:0/49587774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9fd400e040 tx=0x7f9fd4006ee0).stop 2024-01-21T19:59:09.862 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 shutdown_connections 2024-01-21T19:59:09.862 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 --2- 172.21.15.119:0/49587774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4070430 0x7f9fe40fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:09.863 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 >> 172.21.15.119:0/49587774 conn(0x7f9fe40f63c0 msgr2=0x7f9fe40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:09.863 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 shutdown_connections 2024-01-21T19:59:09.863 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:09.869+0000 7f9febe1a700 1 -- 172.21.15.119:0/49587774 wait complete. 2024-01-21T19:59:10.166 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: audit 2024-01-21T19:59:09.103923+0000 mon.smithi119 (mon.0) 8 : audit [INF] from='client.? 172.21.15.119:0/3706096152' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T19:59:10.166 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: audit 2024-01-21T19:59:09.107062+0000 mon.smithi119 (mon.0) 9 : audit [INF] from='client.? 172.21.15.119:0/3706096152' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T19:59:10.214 INFO:teuthology.orchestra.run.smithi119.stderr:Restarting the monitor... 2024-01-21T19:59:10.214 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl restart ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119 2024-01-21T19:59:10.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 systemd[1]: Stopping Ceph mon.smithi119 for 79b26e34-b897-11ee-95b0-87774f69a715... 2024-01-21T19:59:10.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: debug 2024-01-21T19:59:10.292+0000 7f30c4f3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi119 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T19:59:10.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38408]: debug 2024-01-21T19:59:10.292+0000 7f30c4f3e700 -1 mon.smithi119@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T19:59:11.017 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 podman[38730]: ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119 2024-01-21T19:59:11.017 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 systemd[1]: ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119.service: Succeeded. 2024-01-21T19:59:11.017 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 systemd[1]: Stopped Ceph mon.smithi119 for 79b26e34-b897-11ee-95b0-87774f69a715. 2024-01-21T19:59:11.017 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:10 smithi119 systemd[1]: Starting Ceph mon.smithi119 for 79b26e34-b897-11ee-95b0-87774f69a715... 2024-01-21T19:59:11.081 INFO:teuthology.orchestra.run.smithi119.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-21T19:59:11.081 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.085+0000 7fec9416a700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.085+0000 7fec9416a700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.085+0000 7fec9416a700 0 pidfile_write: ignore empty --pid-file 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 bash[38801]: 21f43314bb637f72da5981a57ff2536a88fc91572649a63faf0d5124e25f7753 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 systemd[1]: Started Ceph mon.smithi119 for 79b26e34-b897-11ee-95b0-87774f69a715. 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 0 load: jerasure load: lrc load: isa 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: RocksDB version: 6.8.1 2024-01-21T19:59:11.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: Compile date May 13 2021 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: DB SUMMARY 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: CURRENT file: CURRENT 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi119/store.db dir, Total Num: 1, files: 000004.sst 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi119/store.db: 000006.log size: 92938 ; 2024-01-21T19:59:11.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: Options.error_if_exists: 0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: Options.create_if_missing: 0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.090+0000 7fec9416a700 4 rocksdb: Options.paranoid_checks: 1 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.env: 0x5643c89791c0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.fs: Posix File System 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.info_log: 0x5643c9d05960 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.statistics: (nil) 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.use_fsync: 0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_log_file_size: 0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-21T19:59:11.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.allow_fallocate: 1 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.use_direct_reads: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.db_log_dir: 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi119/store.db 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_subcompactions: 1 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_background_flushes: -1 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-21T19:59:11.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.write_buffer_manager: 0x5643c9d0bfb0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.unordered_write: 0 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-21T19:59:11.343 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.row_cache: None 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.wal_filter: None 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.preserve_deletes: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.two_write_queues: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.atomic_flush: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-21T19:59:11.344 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.log_readahead_size: 0 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_background_jobs: 2 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_background_compactions: -1 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_open_files: -1 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-21T19:59:11.345 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Compression algorithms supported: 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kZSTD supported: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kXpressCompression supported: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kLZ4Compression supported: 1 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kBZip2Compression supported: 0 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kZlibCompression supported: 1 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: kSnappyCompression supported: 1 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi119/store.db/MANIFEST-000005 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.346 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.merge_operator: 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_filter: None 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5643c9c1c128) 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cache_index_and_filter_blocks: 1 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: pin_top_level_index_and_filter: 1 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: index_type: 0 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: data_block_index_type: 0 2024-01-21T19:59:11.347 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: index_shortening: 1 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: data_block_hash_table_util_ratio: 0.750000 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: hash_index_allow_collision: 1 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: checksum: 1 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: no_block_cache: 0 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_cache: 0x5643c9c52f10 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_cache_name: BinnedLRUCache 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_cache_options: 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: capacity : 536870912 2024-01-21T19:59:11.348 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: num_shard_bits : 4 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: strict_capacity_limit : 0 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: high_pri_pool_ratio: 0.000 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_cache_compressed: (nil) 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: persistent_cache: (nil) 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_size: 4096 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_size_deviation: 10 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_restart_interval: 16 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: index_block_restart_interval: 1 2024-01-21T19:59:11.350 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: metadata_block_size: 4096 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: partition_filters: 0 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: use_delta_encoding: 1 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: whole_key_filtering: 1 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: verify_compression: 0 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: read_amp_bytes_per_bit: 0 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: format_version: 2 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: enable_index_compression: 1 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: block_align: 0 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression: NoCompression 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-21T19:59:11.351 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.num_levels: 7 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-21T19:59:11.352 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-21T19:59:11.353 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-21T19:59:11.354 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.table_properties_collectors: 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.inplace_update_support: 0 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.bloom_locality: 0 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.max_successive_merges: 0 2024-01-21T19:59:11.355 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.ttl: 2592000 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi119/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.091+0000 7fec9416a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.092+0000 7fec9416a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705867151093532, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.092+0000 7fec9416a700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.092+0000 7fec9416a700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-21T19:59:11.356 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.093+0000 7fec9416a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705867151094574, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88718, "table_properties": {"data_size": 86833, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79974, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705867151, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.093+0000 7fec9416a700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.094+0000 7fec9416a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705867151095569, "job": 1, "event": "recovery_finished"} 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.094+0000 7fec9416a700 4 rocksdb: DB pointer 0x5643c9d17800 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.095+0000 7fec7af82700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.095+0000 7fec7af82700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** DB Stats ** 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-21T19:59:11.357 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** Compaction Stats [default] ** 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** Compaction Stats [default] ** 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.358 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Cumulative compaction: 0.00 GB write, 23.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Interval compaction: 0.00 GB write, 23.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** Compaction Stats [default] ** 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:11.359 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** Compaction Stats [default] ** 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.5 0.00 0.00 1 0.001 0 0 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(Total Files): cumulative 0, interval 0 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-21T19:59:11.360 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: AddFile(Keys): cumulative 0, interval 0 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Cumulative compaction: 0.00 GB write, 22.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ** File Read Latency Histogram By Level [default] ** 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.095+0000 7fec9416a700 0 starting mon.smithi119 rank 0 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi119 fsid 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.096+0000 7fec9416a700 1 mon.smithi119@-1(???) e1 preinit fsid 79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 0 mon.smithi119@-1(???).mds e1 new map 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 0 mon.smithi119@-1(???).mds e1 print_map 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: e1 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: legacy client fscid: -1 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.361 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: No filesystems configured 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 0 mon.smithi119@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 0 mon.smithi119@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 0 mon.smithi119@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 0 mon.smithi119@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:09.103923+0000 mon.smithi119 (mon.0) 8 : audit [INF] from='client.? 172.21.15.119:0/3706096152' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:09.107062+0000 mon.smithi119 (mon.0) 9 : audit [INF] from='client.? 172.21.15.119:0/3706096152' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.097+0000 7fec9416a700 1 mon.smithi119@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expand map: {default=false} 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta from 'false' to 'false' 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expanded map: {default=false} 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expand map: {default=info} 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta from 'info' to 'info' 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expanded map: {default=info} 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expand map: {default=daemon} 2024-01-21T19:59:11.362 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expand map: {default=debug} 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T19:59:11.098+0000 7fec9416a700 20 expand_channel_meta expanded map: {default=debug} 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:11.114132+0000 mon.smithi119 (mon.0) 1 : cluster [INF] mon.smithi119 is new leader, mons smithi119 in quorum (ranks 0) 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:11.114189+0000 mon.smithi119 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi119=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:11.114233+0000 mon.smithi119 (mon.0) 3 : cluster [DBG] fsmap 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:11.114262+0000 mon.smithi119 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-21T19:59:11.363 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:11.117210+0000 mon.smithi119 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-21T19:59:11.443 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5923fff700 1 Processor -- start 2024-01-21T19:59:11.443 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5923fff700 1 -- start start 2024-01-21T19:59:11.443 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5923fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f59240fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.443 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5923fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240fec20 con 0x7f5924070430 2024-01-21T19:59:11.443 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5922ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f59240fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5922ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f59240fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42182/0 (socket says 172.21.15.119:42182) 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.449+0000 7f5922ffd700 1 -- 172.21.15.119:0/914647766 learned_addr learned my addr 172.21.15.119:0/914647766 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5922ffd700 1 -- 172.21.15.119:0/914647766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240fed60 con 0x7f5924070430 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5922ffd700 1 --2- 172.21.15.119:0/914647766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f59240fe730 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f591400b5f0 tx=0x7f5914006ee0).ready entity=mon.0 client_cookie=a9cb6270a82cc160 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5921ffb700 1 -- 172.21.15.119:0/914647766 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f591400d600 con 0x7f5924070430 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5921ffb700 1 -- 172.21.15.119:0/914647766 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f591400dbd0 con 0x7f5924070430 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5921ffb700 1 -- 172.21.15.119:0/914647766 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5914011c00 con 0x7f5924070430 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 -- 172.21.15.119:0/914647766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 msgr2=0x7f59240fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 --2- 172.21.15.119:0/914647766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f59240fe730 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f591400b5f0 tx=0x7f5914006ee0).stop 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 -- 172.21.15.119:0/914647766 shutdown_connections 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 --2- 172.21.15.119:0/914647766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f59240fe730 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 -- 172.21.15.119:0/914647766 >> 172.21.15.119:0/914647766 conn(0x7f59240f6210 msgr2=0x7f59240f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:11.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 -- 172.21.15.119:0/914647766 shutdown_connections 2024-01-21T19:59:11.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.450+0000 7f5923fff700 1 -- 172.21.15.119:0/914647766 wait complete. 2024-01-21T19:59:11.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5923fff700 1 Processor -- start 2024-01-21T19:59:11.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5923fff700 1 -- start start 2024-01-21T19:59:11.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5923fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f592410a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:11.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5923fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592410ad20 con 0x7f5924070430 2024-01-21T19:59:11.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5922ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f592410a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:11.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5922ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f592410a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42184/0 (socket says 172.21.15.119:42184) 2024-01-21T19:59:11.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.451+0000 7f5922ffd700 1 -- 172.21.15.119:0/2936349175 learned_addr learned my addr 172.21.15.119:0/2936349175 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f5922ffd700 1 -- 172.21.15.119:0/2936349175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591400b200 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f5922ffd700 1 --2- 172.21.15.119:0/2936349175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f592410a830 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f591400b190 tx=0x7f5914006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f591401b410 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f591401b9e0 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592410af20 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f591401a880 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592410b330 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f591400f070 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.452+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5914025b40 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.453+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5924045ad0 con 0x7f5924070430 2024-01-21T19:59:11.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.455+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5914016070 con 0x7f5924070430 2024-01-21T19:59:11.510 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.518+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f5924104750 con 0x7f5924070430 2024-01-21T19:59:11.515 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.523+0000 7f590b7fe700 1 -- 172.21.15.119:0/2936349175 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5914025e20 con 0x7f5924070430 2024-01-21T19:59:11.516 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 msgr2=0x7f592410a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:11.517 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 --2- 172.21.15.119:0/2936349175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f592410a830 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f591400b190 tx=0x7f5914006ee0).stop 2024-01-21T19:59:11.517 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 shutdown_connections 2024-01-21T19:59:11.517 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 --2- 172.21.15.119:0/2936349175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5924070430 0x7f592410a830 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:11.517 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 >> 172.21.15.119:0/2936349175 conn(0x7f59240f6210 msgr2=0x7f59240f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:11.517 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 shutdown_connections 2024-01-21T19:59:11.517 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:11.524+0000 7f5923fff700 1 -- 172.21.15.119:0/2936349175 wait complete. 2024-01-21T19:59:11.920 INFO:teuthology.orchestra.run.smithi119.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-21T19:59:11.920 INFO:teuthology.orchestra.run.smithi119.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-21T19:59:11.921 INFO:teuthology.orchestra.run.smithi119.stderr:Creating mgr... 2024-01-21T19:59:11.921 INFO:teuthology.orchestra.run.smithi119.stderr:Verifying port 9283 ... 2024-01-21T19:59:11.923 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl daemon-reload 2024-01-21T19:59:12.020 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm 2024-01-21T19:59:12.043 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl reset-failed ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm 2024-01-21T19:59:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Failed to reset failed state of unit ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm.service: Unit ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm.service not loaded. 2024-01-21T19:59:12.049 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl enable ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm 2024-01-21T19:59:12.055 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: Created symlink /etc/systemd/system/ceph-79b26e34-b897-11ee-95b0-87774f69a715.target.wants/ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm.service → /etc/systemd/system/ceph-79b26e34-b897-11ee-95b0-87774f69a715@.service. 2024-01-21T19:59:12.137 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715@mgr.smithi119.rqacbm 2024-01-21T19:59:12.441 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:59:12.449 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: disabled 2024-01-21T19:59:12.449 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:59:12.457 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: inactive 2024-01-21T19:59:12.458 INFO:teuthology.orchestra.run.smithi119.stderr:firewalld.service is not enabled 2024-01-21T19:59:12.458 INFO:teuthology.orchestra.run.smithi119.stderr:Not possible to enable service . firewalld.service is not available 2024-01-21T19:59:12.458 INFO:teuthology.orchestra.run.smithi119.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-21T19:59:12.458 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T19:59:12.465 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: disabled 2024-01-21T19:59:12.466 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T19:59:12.473 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: inactive 2024-01-21T19:59:12.474 INFO:teuthology.orchestra.run.smithi119.stderr:firewalld.service is not enabled 2024-01-21T19:59:12.474 INFO:teuthology.orchestra.run.smithi119.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-21T19:59:12.474 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for mgr to start... 2024-01-21T19:59:12.474 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for mgr... 2024-01-21T19:59:12.474 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:59:12.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:11.524901+0000 mon.smithi119 (mon.0) 6 : audit [INF] from='client.? 172.21.15.119:0/2936349175' entity='client.admin' 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.863+0000 7f967cca6700 1 Processor -- start 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.864+0000 7f967cca6700 1 -- start start 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.864+0000 7f967cca6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f96780fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.864+0000 7f967cca6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96780fb490 con 0x7f96780fab80 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f96780fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f96780fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41038/0 (socket says 172.21.15.119:41038) 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967659c700 1 -- 172.21.15.119:0/1730975431 learned_addr learned my addr 172.21.15.119:0/1730975431 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967659c700 1 -- 172.21.15.119:0/1730975431 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96780fb5d0 con 0x7f96780fab80 2024-01-21T19:59:12.857 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967659c700 1 --2- 172.21.15.119:0/1730975431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f96780fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f966800b5f0 tx=0x7f9668006ee0).ready entity=mon.0 client_cookie=9054570760ec5e42 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:12.858 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967559a700 1 -- 172.21.15.119:0/1730975431 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966800d600 con 0x7f96780fab80 2024-01-21T19:59:12.858 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967559a700 1 -- 172.21.15.119:0/1730975431 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f966800dbd0 con 0x7f96780fab80 2024-01-21T19:59:12.858 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.865+0000 7f967559a700 1 -- 172.21.15.119:0/1730975431 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9668011c00 con 0x7f96780fab80 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 -- 172.21.15.119:0/1730975431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 msgr2=0x7f96780fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 --2- 172.21.15.119:0/1730975431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f96780fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f966800b5f0 tx=0x7f9668006ee0).stop 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 -- 172.21.15.119:0/1730975431 shutdown_connections 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 --2- 172.21.15.119:0/1730975431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f96780fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 -- 172.21.15.119:0/1730975431 >> 172.21.15.119:0/1730975431 conn(0x7f96780f63c0 msgr2=0x7f96780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 -- 172.21.15.119:0/1730975431 shutdown_connections 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.866+0000 7f967cca6700 1 -- 172.21.15.119:0/1730975431 wait complete. 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967cca6700 1 Processor -- start 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967cca6700 1 -- start start 2024-01-21T19:59:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967cca6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f967810d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967cca6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678070470 con 0x7f96780fab80 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f967810d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f967810d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41048/0 (socket says 172.21.15.119:41048) 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.867+0000 7f967659c700 1 -- 172.21.15.119:0/3214431231 learned_addr learned my addr 172.21.15.119:0/3214431231 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f967659c700 1 -- 172.21.15.119:0/3214431231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966800b200 con 0x7f96780fab80 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f967659c700 1 --2- 172.21.15.119:0/3214431231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f967810d720 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f966800e040 tx=0x7f9668006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966801b410 con 0x7f96780fab80 2024-01-21T19:59:12.860 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f966801b9e0 con 0x7f96780fab80 2024-01-21T19:59:12.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9678070670 con 0x7f96780fab80 2024-01-21T19:59:12.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966801a820 con 0x7f96780fab80 2024-01-21T19:59:12.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.868+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967810ddd0 con 0x7f96780fab80 2024-01-21T19:59:12.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.869+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f966800f070 con 0x7f96780fab80 2024-01-21T19:59:12.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.869+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9678045ad0 con 0x7f96780fab80 2024-01-21T19:59:12.861 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.869+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f96680257d0 con 0x7f96780fab80 2024-01-21T19:59:12.865 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.873+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9668016070 con 0x7f96780fab80 2024-01-21T19:59:12.935 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.943+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f96780fc890 con 0x7f96780fab80 2024-01-21T19:59:12.936 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.944+0000 7f9666ffd700 1 -- 172.21.15.119:0/3214431231 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f966801aed0 con 0x7f96780fab80 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "fsid": "79b26e34-b897-11ee-95b0-87774f69a715", 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:59:12.937 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 0 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "smithi119" 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.943 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:59:12.944 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:59:07.844943+0000", 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 msgr2=0x7f967810d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 --2- 172.21.15.119:0/3214431231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f967810d720 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f966800e040 tx=0x7f9668006ee0).stop 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 shutdown_connections 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 --2- 172.21.15.119:0/3214431231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96780fab80 0x7f967810d720 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 >> 172.21.15.119:0/3214431231 conn(0x7f96780f63c0 msgr2=0x7f96780f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 shutdown_connections 2024-01-21T19:59:12.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:12.945+0000 7f967cca6700 1 -- 172.21.15.119:0/3214431231 wait complete. 2024-01-21T19:59:13.322 INFO:teuthology.orchestra.run.smithi119.stderr:mgr not available, waiting (1/15)... 2024-01-21T19:59:13.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:12.945395+0000 mon.smithi119 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.119:0/3214431231' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:59:15.324 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:59:15.674 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.681+0000 7f5feb6fe700 1 Processor -- start 2024-01-21T19:59:15.676 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.681+0000 7f5feb6fe700 1 -- start start 2024-01-21T19:59:15.676 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5feb6fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5feb6fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe40fedd0 con 0x7f5fe4070430 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fe949a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fe949a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41052/0 (socket says 172.21.15.119:41052) 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fe949a700 1 -- 172.21.15.119:0/3640438740 learned_addr learned my addr 172.21.15.119:0/3640438740 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fe949a700 1 -- 172.21.15.119:0/3640438740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40fef10 con 0x7f5fe4070430 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fe949a700 1 --2- 172.21.15.119:0/3640438740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe40fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5fe000b5f0 tx=0x7f5fe0006ee0).ready entity=mon.0 client_cookie=db0f334799a61c01 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fd3fff700 1 -- 172.21.15.119:0/3640438740 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe000d600 con 0x7f5fe4070430 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fd3fff700 1 -- 172.21.15.119:0/3640438740 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5fe000dbd0 con 0x7f5fe4070430 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.682+0000 7f5fd3fff700 1 -- 172.21.15.119:0/3640438740 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe0011c00 con 0x7f5fe4070430 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 -- 172.21.15.119:0/3640438740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 msgr2=0x7f5fe40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 --2- 172.21.15.119:0/3640438740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe40fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5fe000b5f0 tx=0x7f5fe0006ee0).stop 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 -- 172.21.15.119:0/3640438740 shutdown_connections 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 --2- 172.21.15.119:0/3640438740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe40fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:15.677 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 -- 172.21.15.119:0/3640438740 >> 172.21.15.119:0/3640438740 conn(0x7f5fe40f63c0 msgr2=0x7f5fe40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 -- 172.21.15.119:0/3640438740 shutdown_connections 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.683+0000 7f5feb6fe700 1 -- 172.21.15.119:0/3640438740 wait complete. 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5feb6fe700 1 Processor -- start 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5feb6fe700 1 -- start start 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5feb6fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5feb6fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe410aef0 con 0x7f5fe4070430 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5fe949a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5fe949a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41060/0 (socket says 172.21.15.119:41060) 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5fe949a700 1 -- 172.21.15.119:0/1722635855 learned_addr learned my addr 172.21.15.119:0/1722635855 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:15.678 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5fe949a700 1 -- 172.21.15.119:0/1722635855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe000b200 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.684+0000 7f5fe949a700 1 --2- 172.21.15.119:0/1722635855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe410aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5fe000b190 tx=0x7f5fe0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe001b410 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5fe001b9e0 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe001a880 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe410b0f0 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe410b500 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5fe000f070 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.685+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5fe0025b70 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.686+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe4045ad0 con 0x7f5fe4070430 2024-01-21T19:59:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.688+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5fe0016070 con 0x7f5fe4070430 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.768+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5fe41047f0 con 0x7f5fe4070430 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.768+0000 7f5fd1ffb700 1 -- 172.21.15.119:0/1722635855 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f5fe0006a70 con 0x7f5fe4070430 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "fsid": "79b26e34-b897-11ee-95b0-87774f69a715", 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:59:15.761 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.762 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:59:15.762 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 0 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "smithi119" 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:59:15.767 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.768 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:59:15.769 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "available": false, 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:59:07.844943+0000", 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 msgr2=0x7f5fe410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:15.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 --2- 172.21.15.119:0/1722635855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe410aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5fe000b190 tx=0x7f5fe0006ee0).stop 2024-01-21T19:59:15.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 shutdown_connections 2024-01-21T19:59:15.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 --2- 172.21.15.119:0/1722635855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fe4070430 0x7f5fe410aa00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:15.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 >> 172.21.15.119:0/1722635855 conn(0x7f5fe40f63c0 msgr2=0x7f5fe40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:15.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 shutdown_connections 2024-01-21T19:59:15.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:15.769+0000 7f5feb6fe700 1 -- 172.21.15.119:0/1722635855 wait complete. 2024-01-21T19:59:16.074 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:15 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:15.769893+0000 mon.smithi119 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.119:0/1722635855' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:59:16.130 INFO:teuthology.orchestra.run.smithi119.stderr:mgr not available, waiting (2/15)... 2024-01-21T19:59:17.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:17.355357+0000 mon.smithi119 (mon.0) 9 : cluster [INF] Activating manager daemon smithi119.rqacbm 2024-01-21T19:59:17.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:17.357931+0000 mon.smithi119 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi119.rqacbm(active, starting, since 0.00268437s) 2024-01-21T19:59:17.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.358785+0000 mon.smithi119 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:59:17.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.358905+0000 mon.smithi119 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:59:17.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.359014+0000 mon.smithi119 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.359145+0000 mon.smithi119 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata", "id": "smithi119"}]: dispatch 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.359452+0000 mon.smithi119 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mgr metadata", "who": "smithi119.rqacbm", "id": "smithi119.rqacbm"}]: dispatch 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:17.373969+0000 mon.smithi119 (mon.0) 16 : cluster [INF] Manager daemon smithi119.rqacbm is now available 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.381291+0000 mon.smithi119 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.385465+0000 mon.smithi119 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.386590+0000 mon.smithi119 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.391138+0000 mon.smithi119 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:17.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:17.392991+0000 mon.smithi119 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.119:0/3521873104' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:18.132 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-21T19:59:18.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.484+0000 7fc43edfb700 1 Processor -- start 2024-01-21T19:59:18.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.484+0000 7fc43edfb700 1 -- start start 2024-01-21T19:59:18.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.484+0000 7fc43edfb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc438100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:18.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.484+0000 7fc43edfb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4380fc730 con 0x7fc4380703d0 2024-01-21T19:59:18.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc43cb97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc438100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:18.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc43cb97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc438100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41128/0 (socket says 172.21.15.119:41128) 2024-01-21T19:59:18.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc43cb97700 1 -- 172.21.15.119:0/3959950956 learned_addr learned my addr 172.21.15.119:0/3959950956 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:18.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc43cb97700 1 -- 172.21.15.119:0/3959950956 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438101230 con 0x7fc4380703d0 2024-01-21T19:59:18.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc43cb97700 1 --2- 172.21.15.119:0/3959950956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc438100d40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc42c00b5f0 tx=0x7fc42c006ee0).ready entity=mon.0 client_cookie=36746d35518aad1f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:18.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc4377fe700 1 -- 172.21.15.119:0/3959950956 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42c00d600 con 0x7fc4380703d0 2024-01-21T19:59:18.483 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc4377fe700 1 -- 172.21.15.119:0/3959950956 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc42c00dbd0 con 0x7fc4380703d0 2024-01-21T19:59:18.483 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.485+0000 7fc4377fe700 1 -- 172.21.15.119:0/3959950956 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42c011c00 con 0x7fc4380703d0 2024-01-21T19:59:18.483 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 -- 172.21.15.119:0/3959950956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 msgr2=0x7fc438100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:18.483 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 --2- 172.21.15.119:0/3959950956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc438100d40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc42c00b5f0 tx=0x7fc42c006ee0).stop 2024-01-21T19:59:18.483 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 -- 172.21.15.119:0/3959950956 shutdown_connections 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 --2- 172.21.15.119:0/3959950956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc438100d40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 -- 172.21.15.119:0/3959950956 >> 172.21.15.119:0/3959950956 conn(0x7fc4380f63c0 msgr2=0x7fc4380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 -- 172.21.15.119:0/3959950956 shutdown_connections 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 -- 172.21.15.119:0/3959950956 wait complete. 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 Processor -- start 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.486+0000 7fc43edfb700 1 -- start start 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43edfb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc43810cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43edfb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43810d0a0 con 0x7fc4380703d0 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43cb97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc43810cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43cb97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc43810cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41140/0 (socket says 172.21.15.119:41140) 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43cb97700 1 -- 172.21.15.119:0/1445796172 learned_addr learned my addr 172.21.15.119:0/1445796172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43cb97700 1 -- 172.21.15.119:0/1445796172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c00b200 con 0x7fc4380703d0 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43cb97700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc43810cbb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc42c00b190 tx=0x7fc42c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:18.484 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42c01b410 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc42c01b9e0 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43810d2a0 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42c01a880 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.487+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43810d6b0 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.488+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fc42c00f070 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.488+0000 7fc4357fa700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc41c02fc50 0x7fc41c032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.488+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc42c041ff0 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.488+0000 7fc437fff700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc41c02fc50 0x7fc41c032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.488+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438059100 con 0x7fc4380703d0 2024-01-21T19:59:18.485 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.488+0000 7fc437fff700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc41c02fc50 0x7fc41c032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc4240096a0 tx=0x7fc424006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:18.486 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.493+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc42c025460 con 0x7fc4380703d0 2024-01-21T19:59:18.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.696+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc438101d90 con 0x7fc4380703d0 2024-01-21T19:59:18.695 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.696+0000 7fc4357fa700 1 -- 172.21.15.119:0/1445796172 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fc42c022310 con 0x7fc4380703d0 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "fsid": "79b26e34-b897-11ee-95b0-87774f69a715", 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "health": { 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "checks": {}, 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mutes": [] 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum": [ 2024-01-21T19:59:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 0 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "smithi119" 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "monmap": { 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osdmap": { 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-21T19:59:18.697 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "pgmap": { 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.698 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "fsmap": { 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "modules": [ 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "iostat", 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "restful" 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ], 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:59:18.699 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "servicemap": { 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "modified": "2024-01-21T19:59:07.844943+0000", 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "services": {} 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: }, 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc41c02fc50 msgr2=0x7fc41c032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc41c02fc50 0x7fc41c032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc4240096a0 tx=0x7fc424006b40).stop 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 msgr2=0x7fc43810cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc43810cbb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc42c00b190 tx=0x7fc42c006ee0).stop 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 shutdown_connections 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc41c02fc50 0x7fc41c032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 --2- 172.21.15.119:0/1445796172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4380703d0 0x7fc43810cbb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:18.700 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 >> 172.21.15.119:0/1445796172 conn(0x7fc4380f63c0 msgr2=0x7fc4380f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:18.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 shutdown_connections 2024-01-21T19:59:18.701 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:18.698+0000 7fc43edfb700 1 -- 172.21.15.119:0/1445796172 wait complete. 2024-01-21T19:59:19.021 INFO:teuthology.orchestra.run.smithi119.stderr:mgr is available 2024-01-21T19:59:19.021 INFO:teuthology.orchestra.run.smithi119.stderr:Enabling cephadm module... 2024-01-21T19:59:19.022 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-21T19:59:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc058bbe700 1 Processor -- start 2024-01-21T19:59:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc058bbe700 1 -- start start 2024-01-21T19:59:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc058bbe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc054070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc058bbe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0540728f0 con 0x7fc0540724d0 2024-01-21T19:59:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc05259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc054070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc05259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc054070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41146/0 (socket says 172.21.15.119:41146) 2024-01-21T19:59:19.383 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.389+0000 7fc05259c700 1 -- 172.21.15.119:0/1527248953 learned_addr learned my addr 172.21.15.119:0/1527248953 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:19.383 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.390+0000 7fc05259c700 1 -- 172.21.15.119:0/1527248953 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054070f60 con 0x7fc0540724d0 2024-01-21T19:59:19.383 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.390+0000 7fc05259c700 1 --2- 172.21.15.119:0/1527248953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc054070a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc04400b5f0 tx=0x7fc044006ee0).ready entity=mon.0 client_cookie=c906db951e5c16c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:19.383 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.390+0000 7fc05159a700 1 -- 172.21.15.119:0/1527248953 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04400d600 con 0x7fc0540724d0 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.390+0000 7fc05159a700 1 -- 172.21.15.119:0/1527248953 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc04400dbd0 con 0x7fc0540724d0 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.390+0000 7fc05159a700 1 -- 172.21.15.119:0/1527248953 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc044011c00 con 0x7fc0540724d0 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 -- 172.21.15.119:0/1527248953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 msgr2=0x7fc054070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 --2- 172.21.15.119:0/1527248953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc054070a70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc04400b5f0 tx=0x7fc044006ee0).stop 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 -- 172.21.15.119:0/1527248953 shutdown_connections 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 --2- 172.21.15.119:0/1527248953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc054070a70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 -- 172.21.15.119:0/1527248953 >> 172.21.15.119:0/1527248953 conn(0x7fc0540f74c0 msgr2=0x7fc0540f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 -- 172.21.15.119:0/1527248953 shutdown_connections 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 -- 172.21.15.119:0/1527248953 wait complete. 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 Processor -- start 2024-01-21T19:59:19.386 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.391+0000 7fc058bbe700 1 -- start start 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc058bbe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc05410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc058bbe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05410d1d0 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc05259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc05410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc05259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc05410cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41158/0 (socket says 172.21.15.119:41158) 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc05259c700 1 -- 172.21.15.119:0/3538062930 learned_addr learned my addr 172.21.15.119:0/3538062930 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc05259c700 1 -- 172.21.15.119:0/3538062930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04400b200 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc05259c700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc05410cce0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fc044011820 tx=0x7fc044006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04401b410 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc04401b9e0 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc04401a8b0 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05410d3d0 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.392+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05410d7e0 con 0x7fc0540724d0 2024-01-21T19:59:19.387 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.393+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fc04400f070 con 0x7fc0540724d0 2024-01-21T19:59:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.393+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc054045ad0 con 0x7fc0540724d0 2024-01-21T19:59:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.393+0000 7fc042ffd700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc03802fc70 0x7fc038032130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.393+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc044041160 con 0x7fc0540724d0 2024-01-21T19:59:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.394+0000 7fc051d9b700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc03802fc70 0x7fc038032130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.394+0000 7fc051d9b700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc03802fc70 0x7fc038032130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc0480096a0 tx=0x7fc048006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:19.389 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.398+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc044019cd0 con 0x7fc0540724d0 2024-01-21T19:59:19.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:19.580+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc054101e00 con 0x7fc0540724d0 2024-01-21T19:59:19.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:18.360857+0000 mon.smithi119 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi119.rqacbm(active, since 1.00561s) 2024-01-21T19:59:19.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:18.698198+0000 mon.smithi119 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.119:0/1445796172' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-21T19:59:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.358+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc04401bc10 con 0x7fc0540724d0 2024-01-21T19:59:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.360+0000 7fc042ffd700 1 -- 172.21.15.119:0/3538062930 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc044006a70 con 0x7fc0540724d0 2024-01-21T19:59:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc03802fc70 msgr2=0x7fc038032130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc03802fc70 0x7fc038032130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc0480096a0 tx=0x7fc048006b40).stop 2024-01-21T19:59:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 msgr2=0x7fc05410cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc05410cce0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fc044011820 tx=0x7fc044006ee0).stop 2024-01-21T19:59:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 shutdown_connections 2024-01-21T19:59:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fc03802fc70 0x7fc038032130 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 --2- 172.21.15.119:0/3538062930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0540724d0 0x7fc05410cce0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.363+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 >> 172.21.15.119:0/3538062930 conn(0x7fc0540f74c0 msgr2=0x7fc0540f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.364+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 shutdown_connections 2024-01-21T19:59:20.357 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:20.364+0000 7fc058bbe700 1 -- 172.21.15.119:0/3538062930 wait complete. 2024-01-21T19:59:20.656 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:19.582506+0000 mon.smithi119 (mon.0) 24 : audit [INF] from='client.? 172.21.15.119:0/3538062930' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-21T19:59:20.729 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.072+0000 7f208ce47700 1 Processor -- start 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.072+0000 7f208ce47700 1 -- start start 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.072+0000 7f208ce47700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.072+0000 7f208ce47700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20880fee30 con 0x7f20880703d0 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41190/0 (socket says 172.21.15.119:41190) 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208659c700 1 -- 172.21.15.119:0/699349417 learned_addr learned my addr 172.21.15.119:0/699349417 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208659c700 1 -- 172.21.15.119:0/699349417 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20880fef70 con 0x7f20880703d0 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208659c700 1 --2- 172.21.15.119:0/699349417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fe940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f207800b5f0 tx=0x7f2078006ee0).ready entity=mon.0 client_cookie=7c46dcdcaf82b69b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208559a700 1 -- 172.21.15.119:0/699349417 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f207800d600 con 0x7f20880703d0 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208559a700 1 -- 172.21.15.119:0/699349417 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f207800dbd0 con 0x7f20880703d0 2024-01-21T19:59:21.066 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.073+0000 7f208559a700 1 -- 172.21.15.119:0/699349417 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2078011c00 con 0x7f20880703d0 2024-01-21T19:59:21.071 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 -- 172.21.15.119:0/699349417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 msgr2=0x7f20880fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 --2- 172.21.15.119:0/699349417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fe940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f207800b5f0 tx=0x7f2078006ee0).stop 2024-01-21T19:59:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 -- 172.21.15.119:0/699349417 shutdown_connections 2024-01-21T19:59:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 --2- 172.21.15.119:0/699349417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fe940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 -- 172.21.15.119:0/699349417 >> 172.21.15.119:0/699349417 conn(0x7f20880f63c0 msgr2=0x7f20880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 -- 172.21.15.119:0/699349417 shutdown_connections 2024-01-21T19:59:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.074+0000 7f208ce47700 1 -- 172.21.15.119:0/699349417 wait complete. 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.075+0000 7f208ce47700 1 Processor -- start 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.075+0000 7f208ce47700 1 -- start start 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.075+0000 7f208ce47700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.075+0000 7f208ce47700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20880fa840 con 0x7f20880703d0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.075+0000 7f208659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.075+0000 7f208659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41194/0 (socket says 172.21.15.119:41194) 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f208659c700 1 -- 172.21.15.119:0/103042728 learned_addr learned my addr 172.21.15.119:0/103042728 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f208659c700 1 -- 172.21.15.119:0/103042728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207800b200 con 0x7f20880703d0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f208659c700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fc3c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f207800e040 tx=0x7f2078006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f207800d9f0 con 0x7f20880703d0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f207801a9f0 con 0x7f20880703d0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f207801b8c0 con 0x7f20880703d0 2024-01-21T19:59:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20880fa9e0 con 0x7f20880703d0 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.076+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20880fadf0 con 0x7f20880703d0 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.077+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f207800f070 con 0x7f20880703d0 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.077+0000 7f2076ffd700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 0x7f206c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.077+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2088045ad0 con 0x7f20880703d0 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.077+0000 7f2085d9b700 1 -- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 msgr2=0x7f206c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.077+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2078041220 con 0x7f20880703d0 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.077+0000 7f2085d9b700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 0x7f206c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.081+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2078019b40 con 0x7f20880703d0 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.278+0000 7f2085d9b700 1 -- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 msgr2=0x7f206c032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.278+0000 7f2085d9b700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 0x7f206c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.278+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2088110a50 con 0x7f20880703d0 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.278+0000 7f2076ffd700 1 -- 172.21.15.119:0/103042728 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f2078016070 con 0x7f20880703d0 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 4, 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "active_name": "smithi119.rqacbm", 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T19:59:21.271 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:21.275 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 msgr2=0x7f206c032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:21.275 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 0x7f206c032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:21.275 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 msgr2=0x7f20880fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:21.275 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fc3c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f207800e040 tx=0x7f2078006ee0).stop 2024-01-21T19:59:21.276 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 shutdown_connections 2024-01-21T19:59:21.276 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7f206c02fdb0 0x7f206c032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:21.276 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 --2- 172.21.15.119:0/103042728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20880703d0 0x7f20880fc3c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:21.276 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 >> 172.21.15.119:0/103042728 conn(0x7f20880f63c0 msgr2=0x7f20880f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:21.276 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 shutdown_connections 2024-01-21T19:59:21.276 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:21.280+0000 7f208ce47700 1 -- 172.21.15.119:0/103042728 wait complete. 2024-01-21T19:59:21.581 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:20.361545+0000 mon.smithi119 (mon.0) 25 : audit [INF] from='client.? 172.21.15.119:0/3538062930' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-21T19:59:21.581 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:20.361618+0000 mon.smithi119 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi119.rqacbm(active, since 3s) 2024-01-21T19:59:21.581 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:21.280008+0000 mon.smithi119 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.119:0/103042728' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T19:59:21.614 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for the mgr to restart... 2024-01-21T19:59:21.615 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for mgr epoch 4... 2024-01-21T19:59:21.615 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T19:59:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.014+0000 7fcd8c7ad700 1 Processor -- start 2024-01-21T19:59:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.014+0000 7fcd8c7ad700 1 -- start start 2024-01-21T19:59:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8c7ad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd840fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8c7ad700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd840fb490 con 0x7fcd840fab80 2024-01-21T19:59:22.015 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8a549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd840fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:22.015 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8a549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd840fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41204/0 (socket says 172.21.15.119:41204) 2024-01-21T19:59:22.015 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8a549700 1 -- 172.21.15.119:0/2001794066 learned_addr learned my addr 172.21.15.119:0/2001794066 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:22.015 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8a549700 1 -- 172.21.15.119:0/2001794066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd840fb5d0 con 0x7fcd840fab80 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.015+0000 7fcd8a549700 1 --2- 172.21.15.119:0/2001794066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd840fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fcd8000b5f0 tx=0x7fcd80006ee0).ready entity=mon.0 client_cookie=108f082ffa894083 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd89547700 1 -- 172.21.15.119:0/2001794066 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd8000d600 con 0x7fcd840fab80 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd89547700 1 -- 172.21.15.119:0/2001794066 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcd8000dbd0 con 0x7fcd840fab80 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd89547700 1 -- 172.21.15.119:0/2001794066 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd80011c00 con 0x7fcd840fab80 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/2001794066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 msgr2=0x7fcd840fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/2001794066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd840fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fcd8000b5f0 tx=0x7fcd80006ee0).stop 2024-01-21T19:59:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/2001794066 shutdown_connections 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/2001794066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd840fafa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/2001794066 >> 172.21.15.119:0/2001794066 conn(0x7fcd840f63c0 msgr2=0x7fcd840f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/2001794066 shutdown_connections 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.016+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/2001794066 wait complete. 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8c7ad700 1 Processor -- start 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8c7ad700 1 -- start start 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8c7ad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd8410cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8c7ad700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8410d0d0 con 0x7fcd840fab80 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8a549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd8410cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8a549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd8410cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41216/0 (socket says 172.21.15.119:41216) 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.017+0000 7fcd8a549700 1 -- 172.21.15.119:0/3547605172 learned_addr learned my addr 172.21.15.119:0/3547605172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd8a549700 1 -- 172.21.15.119:0/3547605172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8000b200 con 0x7fcd840fab80 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd8a549700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd8410cbe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcd8000b1d0 tx=0x7fcd80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd8001b410 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcd8001b9e0 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8410d2d0 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd8001a840 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.018+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8410d6e0 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.019+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcd8000f070 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.019+0000 7fcd72ffd700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.019+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 --> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd84059100 con 0x7fcd6c02fd90 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.019+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcd80041170 con 0x7fcd840fab80 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.019+0000 7fcd89d48700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 msgr2=0x7fcd6c032250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.019+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.220+0000 7fcd89d48700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 msgr2=0x7fcd6c032250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.220+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:22.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.621+0000 7fcd89d48700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 msgr2=0x7fcd6c032250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:22.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:22.621+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:59:23.414 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:23.422+0000 7fcd89d48700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 msgr2=0x7fcd6c032250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:23.414 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:23.422+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:59:25.015 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:25.023+0000 7fcd89d48700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 msgr2=0x7fcd6c032250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1309038004 2024-01-21T19:59:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:25.023+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:59:25.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:25.355+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fcd800257e0 con 0x7fcd840fab80 2024-01-21T19:59:25.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:25.355+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 msgr2=0x7fcd6c032250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:25.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:25.355+0000 7fcd72ffd700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:25.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:25.305436+0000 mon.smithi119 (mon.0) 28 : cluster [INF] Active manager daemon smithi119.rqacbm restarted 2024-01-21T19:59:25.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:25.305815+0000 mon.smithi119 (mon.0) 29 : cluster [INF] Activating manager daemon smithi119.rqacbm 2024-01-21T19:59:25.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:25.307556+0000 mon.smithi119 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-21T19:59:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.364+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fcd8001b740 con 0x7fcd840fab80 2024-01-21T19:59:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.364+0000 7fcd72ffd700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fcd6c0336b0 0x7fcd6c035aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.364+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd84059100 con 0x7fcd6c0336b0 2024-01-21T19:59:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.365+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fcd6c0336b0 0x7fcd6c035aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.365+0000 7fcd89d48700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fcd6c0336b0 0x7fcd6c035aa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcd74006fd0 tx=0x7fcd74006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.366+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fcd84059100 con 0x7fcd6c0336b0 2024-01-21T19:59:26.363 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.372+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcd84045ad0 con 0x7fcd6c0336b0 2024-01-21T19:59:26.363 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd72ffd700 1 -- 172.21.15.119:0/3547605172 <== mgr.14114 v2:172.21.15.119:6800/240399160 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcd84045ad0 con 0x7fcd6c0336b0 2024-01-21T19:59:26.365 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:26.365 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2024-01-21T19:59:26.365 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T19:59:26.365 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:26.365 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fcd6c0336b0 msgr2=0x7fcd6c035aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fcd6c0336b0 0x7fcd6c035aa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fcd74006fd0 tx=0x7fcd74006d70).stop 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 msgr2=0x7fcd8410cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd8410cbe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcd8000b1d0 tx=0x7fcd80006ee0).stop 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 shutdown_connections 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fcd6c0336b0 0x7fcd6c035aa0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd840fab80 0x7fcd8410cbe0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 --2- 172.21.15.119:0/3547605172 >> [v2:172.21.15.119:6800/1309038004,v1:172.21.15.119:6801/1309038004] conn(0x7fcd6c02fd90 0x7fcd6c032250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 >> 172.21.15.119:0/3547605172 conn(0x7fcd840f63c0 msgr2=0x7fcd840ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 shutdown_connections 2024-01-21T19:59:26.366 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:26.373+0000 7fcd8c7ad700 1 -- 172.21.15.119:0/3547605172 wait complete. 2024-01-21T19:59:26.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:25.358390+0000 mon.smithi119 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi119.rqacbm(active, starting, since 0.0526461s) 2024-01-21T19:59:26.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.361368+0000 mon.smithi119 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata", "id": "smithi119"}]: dispatch 2024-01-21T19:59:26.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.361562+0000 mon.smithi119 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mgr metadata", "who": "smithi119.rqacbm", "id": "smithi119.rqacbm"}]: dispatch 2024-01-21T19:59:26.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.361730+0000 mon.smithi119 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:59:26.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.361861+0000 mon.smithi119 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.361991+0000 mon.smithi119 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:25.376080+0000 mon.smithi119 (mon.0) 37 : cluster [INF] Manager daemon smithi119.rqacbm is now available 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.391506+0000 mon.smithi119 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.394247+0000 mon.smithi119 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.399765+0000 mon.smithi119 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.487181+0000 mon.smithi119 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.487830+0000 mon.smithi119 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.489426+0000 mon.smithi119 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.490883+0000 mon.smithi119 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.500388+0000 mon.smithi119 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:59:26.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:25.501505+0000 mon.smithi119 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T19:59:27.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:26.366582+0000 mgr.smithi119.rqacbm (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T19:59:27.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:26.366934+0000 mon.smithi119 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi119.rqacbm(active, since 1.06119s) 2024-01-21T19:59:27.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:26.373351+0000 mgr.smithi119.rqacbm (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T19:59:28.839 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:27.369677+0000 mon.smithi119 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi119.rqacbm(active, since 2s) 2024-01-21T19:59:28.966 INFO:teuthology.orchestra.run.smithi119.stderr:mgr epoch 4 is available 2024-01-21T19:59:28.966 INFO:teuthology.orchestra.run.smithi119.stderr:Setting orchestrator backend to cephadm... 2024-01-21T19:59:28.966 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.293+0000 7f8670084700 1 Processor -- start 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f8670084700 1 -- start start 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f8670084700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f8668070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f8670084700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86680728d0 con 0x7f86680724b0 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f866de20700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f8668070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f866de20700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f8668070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56194/0 (socket says 172.21.15.119:56194) 2024-01-21T19:59:30.464 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f866de20700 1 -- 172.21.15.119:0/1778947815 learned_addr learned my addr 172.21.15.119:0/1778947815 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f866de20700 1 -- 172.21.15.119:0/1778947815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8668070f40 con 0x7f86680724b0 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.294+0000 7f866de20700 1 --2- 172.21.15.119:0/1778947815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f8668070a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f865800b5f0 tx=0x7f8658006ee0).ready entity=mon.0 client_cookie=daac89f6807c758c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f866ce1e700 1 -- 172.21.15.119:0/1778947815 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865800d600 con 0x7f86680724b0 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f866ce1e700 1 -- 172.21.15.119:0/1778947815 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f865800dbd0 con 0x7f86680724b0 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f866ce1e700 1 -- 172.21.15.119:0/1778947815 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8658011c00 con 0x7f86680724b0 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 -- 172.21.15.119:0/1778947815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 msgr2=0x7f8668070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 --2- 172.21.15.119:0/1778947815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f8668070a50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f865800b5f0 tx=0x7f8658006ee0).stop 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 -- 172.21.15.119:0/1778947815 shutdown_connections 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 --2- 172.21.15.119:0/1778947815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f8668070a50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 -- 172.21.15.119:0/1778947815 >> 172.21.15.119:0/1778947815 conn(0x7f86680f74c0 msgr2=0x7f86680f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 -- 172.21.15.119:0/1778947815 shutdown_connections 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.295+0000 7f8670084700 1 -- 172.21.15.119:0/1778947815 wait complete. 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f8670084700 1 Processor -- start 2024-01-21T19:59:30.465 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f8670084700 1 -- start start 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f8670084700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f866810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f8670084700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866810d1c0 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f866de20700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f866810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f866de20700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f866810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56206/0 (socket says 172.21.15.119:56206) 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f866de20700 1 -- 172.21.15.119:0/1309034786 learned_addr learned my addr 172.21.15.119:0/1309034786 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f866de20700 1 -- 172.21.15.119:0/1309034786 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865800b200 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.296+0000 7f866de20700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f866810ccd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f865800b190 tx=0x7f8658006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.297+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865801b410 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.297+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f865801b9e0 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.297+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866810d3c0 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.297+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865801a880 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.297+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866810d7d0 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.297+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f865800f070 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.298+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8668045ad0 con 0x7f86680724b0 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.298+0000 7f86567fc700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f865002fdb0 0x7f8650032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:30.466 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.298+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f86580411e0 con 0x7f86680724b0 2024-01-21T19:59:30.467 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.298+0000 7f866d61f700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f865002fdb0 0x7f8650032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:30.467 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.298+0000 7f866d61f700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f865002fdb0 0x7f8650032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f86640096a0 tx=0x7f8664006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:30.467 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.302+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86580224f0 con 0x7f86680724b0 2024-01-21T19:59:30.472 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.482+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f866810e030 con 0x7f865002fdb0 2024-01-21T19:59:30.478 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.487+0000 7f86567fc700 1 -- 172.21.15.119:0/1309034786 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f866810e030 con 0x7f865002fdb0 2024-01-21T19:59:30.480 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f865002fdb0 msgr2=0x7f8650032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f865002fdb0 0x7f8650032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f86640096a0 tx=0x7f8664006b40).stop 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 msgr2=0x7f866810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f866810ccd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f865800b190 tx=0x7f8658006ee0).stop 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 shutdown_connections 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f865002fdb0 0x7f8650032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 --2- 172.21.15.119:0/1309034786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86680724b0 0x7f866810ccd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 >> 172.21.15.119:0/1309034786 conn(0x7f86680f74c0 msgr2=0x7f86680f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:30.481 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 shutdown_connections 2024-01-21T19:59:30.482 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:30.489+0000 7f8670084700 1 -- 172.21.15.119:0/1309034786 wait complete. 2024-01-21T19:59:31.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:30.483295+0000 mgr.smithi119.rqacbm (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:31.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:30.487189+0000 mon.smithi119 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:31.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:30.491038+0000 mon.smithi119 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:32.587 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.813+0000 7f25433f8700 1 Processor -- start 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.813+0000 7f25433f8700 1 -- start start 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.813+0000 7f25433f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.813+0000 7f25433f8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c0fc730 con 0x7f253c0703d0 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f2541194700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f2541194700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55992/0 (socket says 172.21.15.119:55992) 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f2541194700 1 -- 172.21.15.119:0/2329037400 learned_addr learned my addr 172.21.15.119:0/2329037400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f2541194700 1 -- 172.21.15.119:0/2329037400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c101230 con 0x7f253c0703d0 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f2541194700 1 --2- 172.21.15.119:0/2329037400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c100d40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f253400b5f0 tx=0x7f2534006ee0).ready entity=mon.0 client_cookie=637b5e87630a164f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f252bfff700 1 -- 172.21.15.119:0/2329037400 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f253400d600 con 0x7f253c0703d0 2024-01-21T19:59:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f252bfff700 1 -- 172.21.15.119:0/2329037400 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f253400dbd0 con 0x7f253c0703d0 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.814+0000 7f252bfff700 1 -- 172.21.15.119:0/2329037400 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2534011c00 con 0x7f253c0703d0 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- 172.21.15.119:0/2329037400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 msgr2=0x7f253c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 --2- 172.21.15.119:0/2329037400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c100d40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f253400b5f0 tx=0x7f2534006ee0).stop 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- 172.21.15.119:0/2329037400 shutdown_connections 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 --2- 172.21.15.119:0/2329037400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c100d40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- 172.21.15.119:0/2329037400 >> 172.21.15.119:0/2329037400 conn(0x7f253c0f63c0 msgr2=0x7f253c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- 172.21.15.119:0/2329037400 shutdown_connections 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- 172.21.15.119:0/2329037400 wait complete. 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 Processor -- start 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- start start 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.815+0000 7f25433f8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c10d0a0 con 0x7f253c0703d0 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2541194700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2541194700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55996/0 (socket says 172.21.15.119:55996) 2024-01-21T19:59:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2541194700 1 -- 172.21.15.119:0/3264329425 learned_addr learned my addr 172.21.15.119:0/3264329425 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2541194700 1 -- 172.21.15.119:0/3264329425 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253400b200 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2541194700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c10cbb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f253400b190 tx=0x7f2534006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f253401b410 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f253401b9e0 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f253401a880 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c10d2a0 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.816+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c10d6b0 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.817+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f253400f070 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.817+0000 7f2529ffb700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f252402fdb0 0x7f2524032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.817+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2534041170 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.817+0000 7f2540993700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f252402fdb0 0x7f2524032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.817+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c059100 con 0x7f253c0703d0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.817+0000 7f2540993700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f252402fdb0 0x7f2524032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f252c0096a0 tx=0x7f252c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:33.947 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.822+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2534016020 con 0x7f253c0703d0 2024-01-21T19:59:33.967 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.976+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f253c106af0 con 0x7f252402fdb0 2024-01-21T19:59:33.967 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.976+0000 7f2529ffb700 1 -- 172.21.15.119:0/3264329425 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f253c106af0 con 0x7f252402fdb0 2024-01-21T19:59:33.967 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: value unchanged 2024-01-21T19:59:33.969 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f252402fdb0 msgr2=0x7f2524032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:33.969 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f252402fdb0 0x7f2524032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f252c0096a0 tx=0x7f252c006b40).stop 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 msgr2=0x7f253c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c10cbb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f253400b190 tx=0x7f2534006ee0).stop 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 shutdown_connections 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f252402fdb0 0x7f2524032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 --2- 172.21.15.119:0/3264329425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f253c0703d0 0x7f253c10cbb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 >> 172.21.15.119:0/3264329425 conn(0x7f253c0f63c0 msgr2=0x7f253c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 shutdown_connections 2024-01-21T19:59:33.970 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:33.978+0000 7f25433f8700 1 -- 172.21.15.119:0/3264329425 wait complete. 2024-01-21T19:59:34.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:33.976923+0000 mgr.smithi119.rqacbm (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:35.268 INFO:teuthology.orchestra.run.smithi119.stderr:Generating ssh key... 2024-01-21T19:59:35.269 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-21T19:59:35.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a720f8700 1 Processor -- start 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a720f8700 1 -- start start 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a720f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a720f8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c070790 con 0x7f4a6c070370 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a6b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a6b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56000/0 (socket says 172.21.15.119:56000) 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.599+0000 7f4a6b7fe700 1 -- 172.21.15.119:0/1231316152 learned_addr learned my addr 172.21.15.119:0/1231316152 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a6b7fe700 1 -- 172.21.15.119:0/1231316152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0fee90 con 0x7f4a6c070370 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a6b7fe700 1 --2- 172.21.15.119:0/1231316152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fe9a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f4a5400b5f0 tx=0x7f4a54006ee0).ready entity=mon.0 client_cookie=4ed6c54db10d085e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a6a7fc700 1 -- 172.21.15.119:0/1231316152 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5400d600 con 0x7f4a6c070370 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a6a7fc700 1 -- 172.21.15.119:0/1231316152 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a5400dbd0 con 0x7f4a6c070370 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a6a7fc700 1 -- 172.21.15.119:0/1231316152 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a54011c00 con 0x7f4a6c070370 2024-01-21T19:59:35.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a720f8700 1 -- 172.21.15.119:0/1231316152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 msgr2=0x7f4a6c0fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a720f8700 1 --2- 172.21.15.119:0/1231316152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fe9a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f4a5400b5f0 tx=0x7f4a54006ee0).stop 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a720f8700 1 -- 172.21.15.119:0/1231316152 shutdown_connections 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a720f8700 1 --2- 172.21.15.119:0/1231316152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fe9a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.600+0000 7f4a720f8700 1 -- 172.21.15.119:0/1231316152 >> 172.21.15.119:0/1231316152 conn(0x7f4a6c0f63c0 msgr2=0x7f4a6c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.601+0000 7f4a720f8700 1 -- 172.21.15.119:0/1231316152 shutdown_connections 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.601+0000 7f4a720f8700 1 -- 172.21.15.119:0/1231316152 wait complete. 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.601+0000 7f4a720f8700 1 Processor -- start 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.601+0000 7f4a720f8700 1 -- start start 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.601+0000 7f4a720f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.601+0000 7f4a720f8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a6c0fa8a0 con 0x7f4a6c070370 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a6b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a6b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56010/0 (socket says 172.21.15.119:56010) 2024-01-21T19:59:35.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a6b7fe700 1 -- 172.21.15.119:0/2528520255 learned_addr learned my addr 172.21.15.119:0/2528520255 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a6b7fe700 1 -- 172.21.15.119:0/2528520255 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5400b200 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a6b7fe700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fc420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f4a5400b190 tx=0x7f4a54006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5401b410 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a5401b9e0 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6c0faa40 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a5401a8f0 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.602+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a6c0fae50 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.603+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4a5400f070 con 0x7f4a6c070370 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.603+0000 7f4a63fff700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f4a4c02fdb0 0x7f4a4c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:35.599 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.603+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4a54041260 con 0x7f4a6c070370 2024-01-21T19:59:35.600 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.603+0000 7f4a6affd700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f4a4c02fdb0 0x7f4a4c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:35.600 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.603+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c059100 con 0x7f4a6c070370 2024-01-21T19:59:35.600 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.604+0000 7f4a6affd700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f4a4c02fdb0 0x7f4a4c032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4a5c0096a0 tx=0x7f4a5c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:35.600 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.607+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a54016070 con 0x7f4a6c070370 2024-01-21T19:59:35.748 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:35.757+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a6c0ffb70 con 0x7f4a4c02fdb0 2024-01-21T19:59:36.080 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.089+0000 7f4a63fff700 1 -- 172.21.15.119:0/2528520255 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a6c0ffb70 con 0x7f4a4c02fdb0 2024-01-21T19:59:36.082 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f4a4c02fdb0 msgr2=0x7f4a4c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:36.082 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f4a4c02fdb0 0x7f4a4c032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4a5c0096a0 tx=0x7f4a5c006b40).stop 2024-01-21T19:59:36.082 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 msgr2=0x7f4a6c0fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:36.082 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fc420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f4a5400b190 tx=0x7f4a54006ee0).stop 2024-01-21T19:59:36.082 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 shutdown_connections 2024-01-21T19:59:36.083 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f4a4c02fdb0 0x7f4a4c032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:36.083 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 --2- 172.21.15.119:0/2528520255 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a6c070370 0x7f4a6c0fc420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:36.083 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 >> 172.21.15.119:0/2528520255 conn(0x7f4a6c0f63c0 msgr2=0x7f4a6c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:36.083 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 shutdown_connections 2024-01-21T19:59:36.083 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.091+0000 7f4a720f8700 1 -- 172.21.15.119:0/2528520255 wait complete. 2024-01-21T19:59:36.451 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.776+0000 7f9bdc3e9700 1 Processor -- start 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.776+0000 7f9bdc3e9700 1 -- start start 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.776+0000 7f9bdc3e9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.776+0000 7f9bdc3e9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd40fee10 con 0x7f9bd40703f0 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.776+0000 7f9bda185700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bda185700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56020/0 (socket says 172.21.15.119:56020) 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bda185700 1 -- 172.21.15.119:0/2795358193 learned_addr learned my addr 172.21.15.119:0/2795358193 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bda185700 1 -- 172.21.15.119:0/2795358193 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd40fef50 con 0x7f9bd40703f0 2024-01-21T19:59:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bda185700 1 --2- 172.21.15.119:0/2795358193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd40fe920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9bc400b5f0 tx=0x7f9bc4006ee0).ready entity=mon.0 client_cookie=64682dd9d5953c9b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bd9183700 1 -- 172.21.15.119:0/2795358193 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bc400d600 con 0x7f9bd40703f0 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bd9183700 1 -- 172.21.15.119:0/2795358193 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9bc400dbd0 con 0x7f9bd40703f0 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.777+0000 7f9bd9183700 1 -- 172.21.15.119:0/2795358193 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bc4011c00 con 0x7f9bd40703f0 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/2795358193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 msgr2=0x7f9bd40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 --2- 172.21.15.119:0/2795358193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd40fe920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9bc400b5f0 tx=0x7f9bc4006ee0).stop 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/2795358193 shutdown_connections 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 --2- 172.21.15.119:0/2795358193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd40fe920 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/2795358193 >> 172.21.15.119:0/2795358193 conn(0x7f9bd40f63c0 msgr2=0x7f9bd40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/2795358193 shutdown_connections 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/2795358193 wait complete. 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 Processor -- start 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- start start 2024-01-21T19:59:36.771 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:36.772 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.778+0000 7f9bdc3e9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd410aeb0 con 0x7f9bd40703f0 2024-01-21T19:59:36.772 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bda185700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:36.772 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bda185700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56026/0 (socket says 172.21.15.119:56026) 2024-01-21T19:59:36.772 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bda185700 1 -- 172.21.15.119:0/1585296155 learned_addr learned my addr 172.21.15.119:0/1585296155 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:36.772 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bda185700 1 -- 172.21.15.119:0/1585296155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc400b200 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bda185700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd410a9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9bc400b190 tx=0x7f9bc4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bc401b410 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9bc401b9e0 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bc401a880 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd410b0b0 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.779+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd410b4c0 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.780+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9bc400f070 con 0x7f9bd40703f0 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.780+0000 7f9bc2ffd700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f9bbc02fdb0 0x7f9bbc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:36.773 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.780+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9bc40411a0 con 0x7f9bd40703f0 2024-01-21T19:59:36.774 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.780+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd4045ad0 con 0x7f9bd40703f0 2024-01-21T19:59:36.774 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.780+0000 7f9bd9984700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f9bbc02fdb0 0x7f9bbc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:36.774 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.780+0000 7f9bd9984700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f9bbc02fdb0 0x7f9bbc032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9bd00096a0 tx=0x7f9bd0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:36.775 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.784+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bc4025460 con 0x7f9bd40703f0 2024-01-21T19:59:36.925 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.933+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bd4104900 con 0x7f9bbc02fdb0 2024-01-21T19:59:36.925 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.934+0000 7f9bc2ffd700 1 -- 172.21.15.119:0/1585296155 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f9bd4104900 con 0x7f9bbc02fdb0 2024-01-21T19:59:36.925 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+YY1WDnw6M9Okf9N8Hn2MO6pKuPeL6ZW86vJySGF/Y7m2sOKYeATKpmCFPC2b30AwTofz5mYoAzvDuLoUwWCV8eVK5bEDzDDFv6yRiBxCBfLTdpFPH/9OrGO7v1RszvcWPGivrKdFlXh+qgShKMTVnFausqGtDpb29LCct2P3V6aw4cUABHena3UfmoRdwV+YvPcX0V+0hA3Hd+vAJdqP4RYfCQ4wxg73nsfZAhzqz2/fKq1hrDrWWHqY4IQBl78Fsm+tvCvOcFhR+olLEj80d99mx8HS6J+/xTSdcPU0kze6P7k3s9F4qRPHQ9dcSfMIabhKsiijswARfn1XS3Ulsl7LLwDjdjC3OC86aUmm0A8n089EB+75/Z+0NjRPMPGUpepLgDTRITBFikag+9sercu2r7joFy59ltDSCeC3kwgkHfTh0aPQGul4HH20op4fdg1q4R+jaFYZNZKh2nMuc/sxr061wwTnj0nvQhffwDZPqQkIP5pVHAV1zyKbNuc= ceph-79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T19:59:36.927 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.935+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f9bbc02fdb0 msgr2=0x7f9bbc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:36.927 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.935+0000 7f9bdc3e9700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f9bbc02fdb0 0x7f9bbc032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9bd00096a0 tx=0x7f9bd0006b40).stop 2024-01-21T19:59:36.927 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.935+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 msgr2=0x7f9bd410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:36.927 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.935+0000 7f9bdc3e9700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd410a9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9bc400b190 tx=0x7f9bc4006ee0).stop 2024-01-21T19:59:36.927 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.936+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 shutdown_connections 2024-01-21T19:59:36.928 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.936+0000 7f9bdc3e9700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f9bbc02fdb0 0x7f9bbc032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:36.928 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.936+0000 7f9bdc3e9700 1 --2- 172.21.15.119:0/1585296155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bd40703f0 0x7f9bd410a9c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:36.928 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.936+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 >> 172.21.15.119:0/1585296155 conn(0x7f9bd40f63c0 msgr2=0x7f9bd40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:36.928 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.936+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 shutdown_connections 2024-01-21T19:59:36.928 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:36.936+0000 7f9bdc3e9700 1 -- 172.21.15.119:0/1585296155 wait complete. 2024-01-21T19:59:37.203 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:35.758246+0000 mgr.smithi119.rqacbm (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-21T19:59:37.204 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:36.087265+0000 mon.smithi119 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:37.204 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:36.089033+0000 mon.smithi119 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:37.276 INFO:teuthology.orchestra.run.smithi119.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-21T19:59:37.277 INFO:teuthology.orchestra.run.smithi119.stderr:Adding key to root@localhost authorized_keys... 2024-01-21T19:59:37.277 INFO:teuthology.orchestra.run.smithi119.stderr:Adding host smithi119... 2024-01-21T19:59:37.277 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi119 172.21.15.119 2024-01-21T19:59:37.611 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa737bdc700 1 Processor -- start 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa737bdc700 1 -- start start 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa737bdc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa737bdc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300fedd0 con 0x7fa730070430 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa735978700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa735978700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56036/0 (socket says 172.21.15.119:56036) 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.616+0000 7fa735978700 1 -- 172.21.15.119:0/2057887654 learned_addr learned my addr 172.21.15.119:0/2057887654 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa735978700 1 -- 172.21.15.119:0/2057887654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7300fef10 con 0x7fa730070430 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa735978700 1 --2- 172.21.15.119:0/2057887654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fe8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa72c00b5f0 tx=0x7fa72c006ee0).ready entity=mon.0 client_cookie=5cd1f84f46c3db4a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa734976700 1 -- 172.21.15.119:0/2057887654 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c00d600 con 0x7fa730070430 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa734976700 1 -- 172.21.15.119:0/2057887654 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa72c00dbd0 con 0x7fa730070430 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa734976700 1 -- 172.21.15.119:0/2057887654 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c011c00 con 0x7fa730070430 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa737bdc700 1 -- 172.21.15.119:0/2057887654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 msgr2=0x7fa7300fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:37.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.617+0000 7fa737bdc700 1 --2- 172.21.15.119:0/2057887654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fe8e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa72c00b5f0 tx=0x7fa72c006ee0).stop 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 -- 172.21.15.119:0/2057887654 shutdown_connections 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 --2- 172.21.15.119:0/2057887654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fe8e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 -- 172.21.15.119:0/2057887654 >> 172.21.15.119:0/2057887654 conn(0x7fa7300f63c0 msgr2=0x7fa7300f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 -- 172.21.15.119:0/2057887654 shutdown_connections 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 -- 172.21.15.119:0/2057887654 wait complete. 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 Processor -- start 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 -- start start 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa737bdc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300fa7e0 con 0x7fa730070430 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa735978700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.618+0000 7fa735978700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56040/0 (socket says 172.21.15.119:56040) 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa735978700 1 -- 172.21.15.119:0/661895534 learned_addr learned my addr 172.21.15.119:0/661895534 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa735978700 1 -- 172.21.15.119:0/661895534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c00b200 con 0x7fa730070430 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa735978700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fc360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa72c002f40 tx=0x7fa72c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:37.613 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c011690 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa72c01b880 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7300fa980 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa72c01a6b0 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.619+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7300fad90 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.620+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa72c00f070 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.620+0000 7fa71e7fc700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fa71802fdb0 0x7fa718032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.620+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa72c0411c0 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.620+0000 7fa735177700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fa71802fdb0 0x7fa718032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.620+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa730059100 con 0x7fa730070430 2024-01-21T19:59:37.614 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.620+0000 7fa735177700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fa71802fdb0 0x7fa718032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa7200096a0 tx=0x7fa720006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:37.615 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.624+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa72c016070 con 0x7fa730070430 2024-01-21T19:59:37.784 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:37.792+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}) v1 -- 0x7fa730104860 con 0x7fa71802fdb0 2024-01-21T19:59:38.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:36.934653+0000 mgr.smithi119.rqacbm (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:38.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Added host 'smithi119' 2024-01-21T19:59:38.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.386+0000 7fa71e7fc700 1 -- 172.21.15.119:0/661895534 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fa730104860 con 0x7fa71802fdb0 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.388+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fa71802fdb0 msgr2=0x7fa718032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.388+0000 7fa737bdc700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fa71802fdb0 0x7fa718032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa7200096a0 tx=0x7fa720006b40).stop 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.388+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 msgr2=0x7fa7300fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.388+0000 7fa737bdc700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fc360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa72c002f40 tx=0x7fa72c006ee0).stop 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.389+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 shutdown_connections 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.389+0000 7fa737bdc700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fa71802fdb0 0x7fa718032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:38.380 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.389+0000 7fa737bdc700 1 --2- 172.21.15.119:0/661895534 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa730070430 0x7fa7300fc360 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.389+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 >> 172.21.15.119:0/661895534 conn(0x7fa7300f63c0 msgr2=0x7fa7300f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.389+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 shutdown_connections 2024-01-21T19:59:38.381 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:38.389+0000 7fa737bdc700 1 -- 172.21.15.119:0/661895534 wait complete. 2024-01-21T19:59:38.685 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying mon service with default placement... 2024-01-21T19:59:38.685 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-21T19:59:38.995 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda750df700 1 Processor -- start 2024-01-21T19:59:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda750df700 1 -- start start 2024-01-21T19:59:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda750df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda70100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda750df700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda700fc750 con 0x7fda700703b0 2024-01-21T19:59:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda6ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda70100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda6ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda70100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56050/0 (socket says 172.21.15.119:56050) 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda6ed9d700 1 -- 172.21.15.119:0/2155374085 learned_addr learned my addr 172.21.15.119:0/2155374085 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.004+0000 7fda6ed9d700 1 -- 172.21.15.119:0/2155374085 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda70101250 con 0x7fda700703b0 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.005+0000 7fda6ed9d700 1 --2- 172.21.15.119:0/2155374085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda70100d60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fda6000b5f0 tx=0x7fda60006ee0).ready entity=mon.0 client_cookie=2abced668818c244 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.005+0000 7fda6dd9b700 1 -- 172.21.15.119:0/2155374085 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6000d600 con 0x7fda700703b0 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.005+0000 7fda6dd9b700 1 -- 172.21.15.119:0/2155374085 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fda6000dbd0 con 0x7fda700703b0 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.005+0000 7fda6dd9b700 1 -- 172.21.15.119:0/2155374085 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda60011c00 con 0x7fda700703b0 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.005+0000 7fda750df700 1 -- 172.21.15.119:0/2155374085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 msgr2=0x7fda70100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.005+0000 7fda750df700 1 --2- 172.21.15.119:0/2155374085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda70100d60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fda6000b5f0 tx=0x7fda60006ee0).stop 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.006+0000 7fda750df700 1 -- 172.21.15.119:0/2155374085 shutdown_connections 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.006+0000 7fda750df700 1 --2- 172.21.15.119:0/2155374085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda70100d60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.000 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.006+0000 7fda750df700 1 -- 172.21.15.119:0/2155374085 >> 172.21.15.119:0/2155374085 conn(0x7fda700f63c0 msgr2=0x7fda700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.006+0000 7fda750df700 1 -- 172.21.15.119:0/2155374085 shutdown_connections 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.006+0000 7fda750df700 1 -- 172.21.15.119:0/2155374085 wait complete. 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.006+0000 7fda750df700 1 Processor -- start 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda750df700 1 -- start start 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda750df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda7010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda750df700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7010d100 con 0x7fda700703b0 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda6ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda7010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda6ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda7010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56060/0 (socket says 172.21.15.119:56060) 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda6ed9d700 1 -- 172.21.15.119:0/210664272 learned_addr learned my addr 172.21.15.119:0/210664272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda6ed9d700 1 -- 172.21.15.119:0/210664272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6000b200 con 0x7fda700703b0 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda6ed9d700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda7010cc10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fda6000b190 tx=0x7fda60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6001b410 con 0x7fda700703b0 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fda6001b9e0 con 0x7fda700703b0 2024-01-21T19:59:39.001 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7010d300 con 0x7fda700703b0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6001a880 con 0x7fda700703b0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.007+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7010d710 con 0x7fda700703b0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.008+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fda6000f070 con 0x7fda700703b0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.008+0000 7fda577fe700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fda5c02fd50 0x7fda5c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.008+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda70045ad0 con 0x7fda700703b0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.008+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fda600412a0 con 0x7fda700703b0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.008+0000 7fda6e59c700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fda5c02fd50 0x7fda5c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.002 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.009+0000 7fda6e59c700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fda5c02fd50 0x7fda5c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fda640096a0 tx=0x7fda64006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.003 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.013+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda60010360 con 0x7fda700703b0 2024-01-21T19:59:39.179 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.188+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fda7010dff0 con 0x7fda5c02fd50 2024-01-21T19:59:39.183 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.192+0000 7fda577fe700 1 -- 172.21.15.119:0/210664272 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fda7010dff0 con 0x7fda5c02fd50 2024-01-21T19:59:39.183 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fda5c02fd50 msgr2=0x7fda5c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fda5c02fd50 0x7fda5c032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fda640096a0 tx=0x7fda64006b40).stop 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 msgr2=0x7fda7010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda7010cc10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fda6000b190 tx=0x7fda60006ee0).stop 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 shutdown_connections 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fda5c02fd50 0x7fda5c032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 --2- 172.21.15.119:0/210664272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda700703b0 0x7fda7010cc10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 >> 172.21.15.119:0/210664272 conn(0x7fda700f63c0 msgr2=0x7fda700f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 shutdown_connections 2024-01-21T19:59:39.186 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.194+0000 7fda750df700 1 -- 172.21.15.119:0/210664272 wait complete. 2024-01-21T19:59:39.486 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:37.794087+0000 mgr.smithi119.rqacbm (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:39.486 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:38.386103+0000 mon.smithi119 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:39.487 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:38.386407+0000 mgr.smithi119.rqacbm (mgr.14114) 9 : cephadm [INF] Added host smithi119 2024-01-21T19:59:39.487 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:38.386640+0000 mon.smithi119 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:39.487 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:39.192763+0000 mon.smithi119 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying mgr service with default placement... 2024-01-21T19:59:39.543 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-21T19:59:39.846 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.852+0000 7f01cf30b700 1 Processor -- start 2024-01-21T19:59:39.846 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.852+0000 7f01cf30b700 1 -- start start 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.852+0000 7f01cf30b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c8100ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.852+0000 7f01cf30b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c80fc6d0 con 0x7f01c8070430 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.852+0000 7f01cd0a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c8100ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01cd0a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c8100ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56062/0 (socket says 172.21.15.119:56062) 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01cd0a7700 1 -- 172.21.15.119:0/44013367 learned_addr learned my addr 172.21.15.119:0/44013367 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01cd0a7700 1 -- 172.21.15.119:0/44013367 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c81011d0 con 0x7f01c8070430 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01cd0a7700 1 --2- 172.21.15.119:0/44013367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c8100ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f01b800b5f0 tx=0x7f01b8006ee0).ready entity=mon.0 client_cookie=87ce5c0edac45e89 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01b7fff700 1 -- 172.21.15.119:0/44013367 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b800d600 con 0x7f01c8070430 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01b7fff700 1 -- 172.21.15.119:0/44013367 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01b800dbd0 con 0x7f01c8070430 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.853+0000 7f01b7fff700 1 -- 172.21.15.119:0/44013367 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b8011c00 con 0x7f01c8070430 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- 172.21.15.119:0/44013367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 msgr2=0x7f01c8100ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 --2- 172.21.15.119:0/44013367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c8100ce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f01b800b5f0 tx=0x7f01b8006ee0).stop 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- 172.21.15.119:0/44013367 shutdown_connections 2024-01-21T19:59:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 --2- 172.21.15.119:0/44013367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c8100ce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- 172.21.15.119:0/44013367 >> 172.21.15.119:0/44013367 conn(0x7f01c80f63c0 msgr2=0x7f01c80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- 172.21.15.119:0/44013367 shutdown_connections 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- 172.21.15.119:0/44013367 wait complete. 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 Processor -- start 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- start start 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c810cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.854+0000 7f01cf30b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c810d0c0 con 0x7f01c8070430 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cd0a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c810cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cd0a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c810cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56068/0 (socket says 172.21.15.119:56068) 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cd0a7700 1 -- 172.21.15.119:0/1186431778 learned_addr learned my addr 172.21.15.119:0/1186431778 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cd0a7700 1 -- 172.21.15.119:0/1186431778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b800b200 con 0x7f01c8070430 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cd0a7700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c810cbd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f01b800b190 tx=0x7f01b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b801b410 con 0x7f01c8070430 2024-01-21T19:59:39.848 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01b801b9e0 con 0x7f01c8070430 2024-01-21T19:59:39.849 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c810d2c0 con 0x7f01c8070430 2024-01-21T19:59:39.849 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b801a880 con 0x7f01c8070430 2024-01-21T19:59:39.849 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.855+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c810d6d0 con 0x7f01c8070430 2024-01-21T19:59:39.850 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.856+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f01b800f070 con 0x7f01c8070430 2024-01-21T19:59:39.850 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.856+0000 7f01b5ffb700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f01b002fdb0 0x7f01b0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:39.850 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.856+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f01b8042060 con 0x7f01c8070430 2024-01-21T19:59:39.850 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.856+0000 7f01cc8a6700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f01b002fdb0 0x7f01b0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:39.850 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.856+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c8059100 con 0x7f01c8070430 2024-01-21T19:59:39.850 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.857+0000 7f01cc8a6700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f01b002fdb0 0x7f01b0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f01c40096a0 tx=0x7f01c4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:39.852 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:39.860+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01b8025460 con 0x7f01c8070430 2024-01-21T19:59:40.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.034+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f01c8106a60 con 0x7f01b002fdb0 2024-01-21T19:59:40.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.038+0000 7f01b5ffb700 1 -- 172.21.15.119:0/1186431778 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f01c8106a60 con 0x7f01b002fdb0 2024-01-21T19:59:40.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-21T19:59:40.031 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f01b002fdb0 msgr2=0x7f01b0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f01b002fdb0 0x7f01b0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f01c40096a0 tx=0x7f01c4006b40).stop 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 msgr2=0x7f01c810cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c810cbd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f01b800b190 tx=0x7f01b8006ee0).stop 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 shutdown_connections 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f01b002fdb0 0x7f01b0032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 --2- 172.21.15.119:0/1186431778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f01c8070430 0x7f01c810cbd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 >> 172.21.15.119:0/1186431778 conn(0x7f01c80f63c0 msgr2=0x7f01c80f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 shutdown_connections 2024-01-21T19:59:40.032 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.040+0000 7f01cf30b700 1 -- 172.21.15.119:0/1186431778 wait complete. 2024-01-21T19:59:40.359 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying crash service with default placement... 2024-01-21T19:59:40.359 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-21T19:59:40.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:39.189684+0000 mgr.smithi119.rqacbm (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:40.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:39.190368+0000 mgr.smithi119.rqacbm (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-21T19:59:40.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:40.038489+0000 mon.smithi119 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.713+0000 7fd230275700 1 Processor -- start 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.713+0000 7fd230275700 1 -- start start 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.713+0000 7fd230275700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd230275700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2280fee10 con 0x7fd2280703f0 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22e011700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22e011700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56070/0 (socket says 172.21.15.119:56070) 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22e011700 1 -- 172.21.15.119:0/3055000338 learned_addr learned my addr 172.21.15.119:0/3055000338 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22e011700 1 -- 172.21.15.119:0/3055000338 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2280fef50 con 0x7fd2280703f0 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22e011700 1 --2- 172.21.15.119:0/3055000338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd21800b5f0 tx=0x7fd218006ee0).ready entity=mon.0 client_cookie=d63d1fe635019860 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22d00f700 1 -- 172.21.15.119:0/3055000338 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd21800d600 con 0x7fd2280703f0 2024-01-21T19:59:40.707 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22d00f700 1 -- 172.21.15.119:0/3055000338 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd21800dbd0 con 0x7fd2280703f0 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.714+0000 7fd22d00f700 1 -- 172.21.15.119:0/3055000338 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd218011c00 con 0x7fd2280703f0 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 -- 172.21.15.119:0/3055000338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 msgr2=0x7fd2280fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 --2- 172.21.15.119:0/3055000338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd21800b5f0 tx=0x7fd218006ee0).stop 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 -- 172.21.15.119:0/3055000338 shutdown_connections 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 --2- 172.21.15.119:0/3055000338 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fe920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 -- 172.21.15.119:0/3055000338 >> 172.21.15.119:0/3055000338 conn(0x7fd2280f63c0 msgr2=0x7fd2280f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 -- 172.21.15.119:0/3055000338 shutdown_connections 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 -- 172.21.15.119:0/3055000338 wait complete. 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 Processor -- start 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.715+0000 7fd230275700 1 -- start start 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd230275700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd230275700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2280fa820 con 0x7fd2280703f0 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd22e011700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:40.708 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd22e011700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56072/0 (socket says 172.21.15.119:56072) 2024-01-21T19:59:40.709 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd22e011700 1 -- 172.21.15.119:0/2934204800 learned_addr learned my addr 172.21.15.119:0/2934204800 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:40.709 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd22e011700 1 -- 172.21.15.119:0/2934204800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21800b200 con 0x7fd2280703f0 2024-01-21T19:59:40.709 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd22e011700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fc330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd21800e040 tx=0x7fd218006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd21800d9f0 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd21801a9f0 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2280faa20 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd21801b8c0 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.716+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2280fae30 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.717+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd21800f070 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.717+0000 7fd216ffd700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fd21002fd50 0x7fd210032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.717+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd21801ab60 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.717+0000 7fd22d810700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fd21002fd50 0x7fd210032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.717+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd228045ad0 con 0x7fd2280703f0 2024-01-21T19:59:40.710 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.718+0000 7fd22d810700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fd21002fd50 0x7fd210032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd2240096a0 tx=0x7fd224006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:40.712 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.721+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd218016070 con 0x7fd2280703f0 2024-01-21T19:59:40.886 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.895+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fd228104850 con 0x7fd21002fd50 2024-01-21T19:59:40.890 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.899+0000 7fd216ffd700 1 -- 172.21.15.119:0/2934204800 <== mgr.14114 v2:172.21.15.119:6800/240399160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fd228104850 con 0x7fd21002fd50 2024-01-21T19:59:40.890 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-21T19:59:40.892 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fd21002fd50 msgr2=0x7fd210032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:40.892 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fd21002fd50 0x7fd210032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd2240096a0 tx=0x7fd224006b40).stop 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 msgr2=0x7fd2280fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fc330 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd21800e040 tx=0x7fd218006ee0).stop 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 shutdown_connections 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fd21002fd50 0x7fd210032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 --2- 172.21.15.119:0/2934204800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2280703f0 0x7fd2280fc330 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 >> 172.21.15.119:0/2934204800 conn(0x7fd2280f63c0 msgr2=0x7fd2280f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 shutdown_connections 2024-01-21T19:59:40.893 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:40.901+0000 7fd230275700 1 -- 172.21.15.119:0/2934204800 wait complete. 2024-01-21T19:59:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:Enabling mgr prometheus module... 2024-01-21T19:59:41.259 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-21T19:59:41.587 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.594+0000 7f51a997f700 1 Processor -- start 2024-01-21T19:59:41.587 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.594+0000 7f51a997f700 1 -- start start 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.594+0000 7f51a997f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a4100d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.594+0000 7f51a997f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a40fc740 con 0x7f51a4070440 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a4100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a4100d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56076/0 (socket says 172.21.15.119:56076) 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a2ffd700 1 -- 172.21.15.119:0/1265010420 learned_addr learned my addr 172.21.15.119:0/1265010420 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a2ffd700 1 -- 172.21.15.119:0/1265010420 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a4101240 con 0x7f51a4070440 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a2ffd700 1 --2- 172.21.15.119:0/1265010420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a4100d50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f519800b5f0 tx=0x7f5198006ee0).ready entity=mon.0 client_cookie=216a8ed62fe5716f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a1ffb700 1 -- 172.21.15.119:0/1265010420 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f519800d600 con 0x7f51a4070440 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a1ffb700 1 -- 172.21.15.119:0/1265010420 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f519800dbd0 con 0x7f51a4070440 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.595+0000 7f51a1ffb700 1 -- 172.21.15.119:0/1265010420 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5198011c00 con 0x7f51a4070440 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 -- 172.21.15.119:0/1265010420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 msgr2=0x7f51a4100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:41.588 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 --2- 172.21.15.119:0/1265010420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a4100d50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f519800b5f0 tx=0x7f5198006ee0).stop 2024-01-21T19:59:41.589 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 -- 172.21.15.119:0/1265010420 shutdown_connections 2024-01-21T19:59:41.589 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 --2- 172.21.15.119:0/1265010420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a4100d50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:41.589 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 -- 172.21.15.119:0/1265010420 >> 172.21.15.119:0/1265010420 conn(0x7f51a40f63b0 msgr2=0x7f51a40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:41.589 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 -- 172.21.15.119:0/1265010420 shutdown_connections 2024-01-21T19:59:41.589 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.596+0000 7f51a997f700 1 -- 172.21.15.119:0/1265010420 wait complete. 2024-01-21T19:59:41.589 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a997f700 1 Processor -- start 2024-01-21T19:59:41.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:40.035687+0000 mgr.smithi119.rqacbm (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:41.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:40.036179+0000 mgr.smithi119.rqacbm (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-21T19:59:41.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:40.899715+0000 mon.smithi119 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.119:0/3303652597' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:41.591 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a997f700 1 -- start start 2024-01-21T19:59:41.591 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a997f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a410cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:41.591 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a997f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a410d0f0 con 0x7f51a4070440 2024-01-21T19:59:41.591 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a410cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:41.591 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a410cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56088/0 (socket says 172.21.15.119:56088) 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a2ffd700 1 -- 172.21.15.119:0/3115016024 learned_addr learned my addr 172.21.15.119:0/3115016024 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a2ffd700 1 -- 172.21.15.119:0/3115016024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519800b200 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.597+0000 7f51a2ffd700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a410cc00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f519800e040 tx=0x7f5198006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.598+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f519800d9f0 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.598+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f519801a9f0 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.598+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a410d2f0 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.598+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f519801b8c0 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.598+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a410d700 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.598+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f519800f070 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.599+0000 7f518b7fe700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f519002fdb0 0x7f5190032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.599+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5198041300 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.599+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4045ad0 con 0x7f51a4070440 2024-01-21T19:59:41.592 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.599+0000 7f51a27fc700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f519002fdb0 0x7f5190032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:41.593 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.599+0000 7f51a27fc700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f519002fdb0 0x7f5190032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f51940096a0 tx=0x7f5194006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:41.594 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.603+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5198025460 con 0x7f51a4070440 2024-01-21T19:59:41.778 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:41.787+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f51a4106a00 con 0x7f51a4070440 2024-01-21T19:59:42.379 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.388+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f51980257e0 con 0x7f51a4070440 2024-01-21T19:59:42.381 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.390+0000 7f518b7fe700 1 -- 172.21.15.119:0/3115016024 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f5198022310 con 0x7f51a4070440 2024-01-21T19:59:42.384 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.392+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f519002fdb0 msgr2=0x7f5190032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:42.384 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.392+0000 7f51a997f700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f519002fdb0 0x7f5190032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f51940096a0 tx=0x7f5194006b40).stop 2024-01-21T19:59:42.384 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.392+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 msgr2=0x7f51a410cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:42.384 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.392+0000 7f51a997f700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a410cc00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f519800e040 tx=0x7f5198006ee0).stop 2024-01-21T19:59:42.384 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.393+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 shutdown_connections 2024-01-21T19:59:42.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.393+0000 7f51a997f700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7f519002fdb0 0x7f5190032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:42.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.393+0000 7f51a997f700 1 --2- 172.21.15.119:0/3115016024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51a4070440 0x7f51a410cc00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:42.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.393+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 >> 172.21.15.119:0/3115016024 conn(0x7f51a40f63b0 msgr2=0x7f51a40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:42.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.393+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 shutdown_connections 2024-01-21T19:59:42.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:42.393+0000 7f51a997f700 1 -- 172.21.15.119:0/3115016024 wait complete. 2024-01-21T19:59:42.693 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:40.896784+0000 mgr.smithi119.rqacbm (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:42.693 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:40.897400+0000 mgr.smithi119.rqacbm (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-21T19:59:42.693 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:41.788692+0000 mon.smithi119 (mon.0) 58 : audit [INF] from='client.? 172.21.15.119:0/3115016024' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-21T19:59:42.734 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying prometheus service with default placement... 2024-01-21T19:59:42.734 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-21T19:59:43.098 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.106+0000 7fe40cfb1700 1 Processor -- start 2024-01-21T19:59:43.098 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40cfb1700 1 -- start start 2024-01-21T19:59:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40cfb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe4080fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40cfb1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080fedd0 con 0x7fe408070430 2024-01-21T19:59:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe4080fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe4080fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44390/0 (socket says 172.21.15.119:44390) 2024-01-21T19:59:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40659c700 1 -- 172.21.15.119:0/1713736622 learned_addr learned my addr 172.21.15.119:0/1713736622 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40659c700 1 -- 172.21.15.119:0/1713736622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080fef10 con 0x7fe408070430 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.107+0000 7fe40659c700 1 --2- 172.21.15.119:0/1713736622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe4080fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe3f800b5f0 tx=0x7fe3f8006ee0).ready entity=mon.0 client_cookie=fe0718310f722984 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40559a700 1 -- 172.21.15.119:0/1713736622 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f800d600 con 0x7fe408070430 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40559a700 1 -- 172.21.15.119:0/1713736622 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe3f800dbd0 con 0x7fe408070430 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40559a700 1 -- 172.21.15.119:0/1713736622 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f8011c00 con 0x7fe408070430 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1713736622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 msgr2=0x7fe4080fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1713736622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe4080fe8e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe3f800b5f0 tx=0x7fe3f8006ee0).stop 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1713736622 shutdown_connections 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1713736622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe4080fe8e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1713736622 >> 172.21.15.119:0/1713736622 conn(0x7fe4080f63c0 msgr2=0x7fe4080f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1713736622 shutdown_connections 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.108+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1713736622 wait complete. 2024-01-21T19:59:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40cfb1700 1 Processor -- start 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40cfb1700 1 -- start start 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40cfb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe40810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40cfb1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40810af30 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe40810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe40810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44392/0 (socket says 172.21.15.119:44392) 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40659c700 1 -- 172.21.15.119:0/1290355669 learned_addr learned my addr 172.21.15.119:0/1290355669 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.109+0000 7fe40659c700 1 -- 172.21.15.119:0/1290355669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f800b200 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe40659c700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe40810aa40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe3f800e040 tx=0x7fe3f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f800d9f0 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe3f801a9f0 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40810b130 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f801b8c0 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40810b540 con 0x7fe408070430 2024-01-21T19:59:43.104 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.110+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fe3f800f070 con 0x7fe408070430 2024-01-21T19:59:43.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.111+0000 7fe3f6ffd700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:43.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.111+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe3f80421a0 con 0x7fe408070430 2024-01-21T19:59:43.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.111+0000 7fe405d9b700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 msgr2=0x7fe3ec0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/240399160 2024-01-21T19:59:43.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.111+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:43.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.111+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe408059100 con 0x7fe408070430 2024-01-21T19:59:43.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.115+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3f8016070 con 0x7fe408070430 2024-01-21T19:59:43.289 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.298+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 --> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fe408104890 con 0x7fe3ec02fe00 2024-01-21T19:59:43.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.312+0000 7fe405d9b700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 msgr2=0x7fe3ec0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/240399160 2024-01-21T19:59:43.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.312+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:43.704 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.713+0000 7fe405d9b700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 msgr2=0x7fe3ec0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/240399160 2024-01-21T19:59:43.704 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:43.713+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:59:43.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:42.390770+0000 mon.smithi119 (mon.0) 59 : audit [INF] from='client.? 172.21.15.119:0/3115016024' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-21T19:59:43.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:42.390825+0000 mon.smithi119 (mon.0) 60 : cluster [DBG] mgrmap e8: smithi119.rqacbm(active, since 17s) 2024-01-21T19:59:44.505 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:44.514+0000 7fe405d9b700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 msgr2=0x7fe3ec0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/240399160 2024-01-21T19:59:44.505 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:44.514+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:59:46.106 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:46.114+0000 7fe405d9b700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 msgr2=0x7fe3ec0322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/240399160 2024-01-21T19:59:46.106 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:46.114+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:59:47.371 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:47.378+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fe3f80257e0 con 0x7fe408070430 2024-01-21T19:59:47.372 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:47.378+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 msgr2=0x7fe3ec0322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:47.372 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:47.378+0000 7fe3f6ffd700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:47.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:47.328223+0000 mon.smithi119 (mon.0) 61 : cluster [INF] Active manager daemon smithi119.rqacbm restarted 2024-01-21T19:59:47.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:47.328600+0000 mon.smithi119 (mon.0) 62 : cluster [INF] Activating manager daemon smithi119.rqacbm 2024-01-21T19:59:47.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:47.330465+0000 mon.smithi119 (mon.0) 63 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-21T19:59:48.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.385+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fe3f8006020 con 0x7fe408070430 2024-01-21T19:59:48.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.385+0000 7fe3f6ffd700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe3ec033c20 0x7fe3ec036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:48.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.385+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 --> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fe408104890 con 0x7fe3ec033c20 2024-01-21T19:59:48.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.385+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe3ec033c20 0x7fe3ec036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:48.378 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.386+0000 7fe405d9b700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe3ec033c20 0x7fe3ec036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fe3fc006fd0 tx=0x7fe3fc006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:48.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.390+0000 7fe3f6ffd700 1 -- 172.21.15.119:0/1290355669 <== mgr.14142 v2:172.21.15.119:6800/2754458491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fe408104890 con 0x7fe3ec033c20 2024-01-21T19:59:48.382 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-21T19:59:48.384 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe3ec033c20 msgr2=0x7fe3ec036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe3ec033c20 0x7fe3ec036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fe3fc006fd0 tx=0x7fe3fc006d30).stop 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 msgr2=0x7fe40810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe40810aa40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe3f800e040 tx=0x7fe3f8006ee0).stop 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 shutdown_connections 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe3ec033c20 0x7fe3ec036010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe408070430 0x7fe40810aa40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 --2- 172.21.15.119:0/1290355669 >> [v2:172.21.15.119:6800/240399160,v1:172.21.15.119:6801/240399160] conn(0x7fe3ec02fe00 0x7fe3ec0322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 >> 172.21.15.119:0/1290355669 conn(0x7fe4080f63c0 msgr2=0x7fe4080f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 shutdown_connections 2024-01-21T19:59:48.385 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:48.392+0000 7fe40cfb1700 1 -- 172.21.15.119:0/1290355669 wait complete. 2024-01-21T19:59:48.668 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:47.381070+0000 mon.smithi119 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi119.rqacbm(active, starting, since 0.0525485s) 2024-01-21T19:59:48.668 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.381975+0000 mon.smithi119 (mon.0) 65 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata", "id": "smithi119"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.384189+0000 mon.smithi119 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mgr metadata", "who": "smithi119.rqacbm", "id": "smithi119.rqacbm"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.384390+0000 mon.smithi119 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.384542+0000 mon.smithi119 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.384676+0000 mon.smithi119 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:47.397943+0000 mon.smithi119 (mon.0) 70 : cluster [INF] Manager daemon smithi119.rqacbm is now available 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.493571+0000 mon.smithi119 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.495277+0000 mon.smithi119 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.509135+0000 mon.smithi119 (mon.0) 73 : audit [INF] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:59:48.669 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:47.511904+0000 mon.smithi119 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T19:59:48.687 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying grafana service with default placement... 2024-01-21T19:59:48.687 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.029+0000 7f8e965d4700 1 Processor -- start 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.029+0000 7f8e965d4700 1 -- start start 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.029+0000 7f8e965d4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e900fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.029+0000 7f8e965d4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e900fee10 con 0x7f8e900703f0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.029+0000 7f8e8ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e900fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e900fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44460/0 (socket says 172.21.15.119:44460) 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8ffff700 1 -- 172.21.15.119:0/816109128 learned_addr learned my addr 172.21.15.119:0/816109128 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8ffff700 1 -- 172.21.15.119:0/816109128 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900fef50 con 0x7f8e900703f0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8ffff700 1 --2- 172.21.15.119:0/816109128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e900fe920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8e8400b5f0 tx=0x7f8e84006ee0).ready entity=mon.0 client_cookie=68a80a0c3db9841 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8effd700 1 -- 172.21.15.119:0/816109128 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e8400d600 con 0x7f8e900703f0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8effd700 1 -- 172.21.15.119:0/816109128 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e8400dbd0 con 0x7f8e900703f0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.030+0000 7f8e8effd700 1 -- 172.21.15.119:0/816109128 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e84011c00 con 0x7f8e900703f0 2024-01-21T19:59:49.025 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 -- 172.21.15.119:0/816109128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 msgr2=0x7f8e900fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 --2- 172.21.15.119:0/816109128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e900fe920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f8e8400b5f0 tx=0x7f8e84006ee0).stop 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 -- 172.21.15.119:0/816109128 shutdown_connections 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 --2- 172.21.15.119:0/816109128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e900fe920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 -- 172.21.15.119:0/816109128 >> 172.21.15.119:0/816109128 conn(0x7f8e900f63c0 msgr2=0x7f8e900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 -- 172.21.15.119:0/816109128 shutdown_connections 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 -- 172.21.15.119:0/816109128 wait complete. 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 Processor -- start 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.031+0000 7f8e965d4700 1 -- start start 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e965d4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e9010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e965d4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9010af40 con 0x7f8e900703f0 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e9010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e9010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44470/0 (socket says 172.21.15.119:44470) 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8ffff700 1 -- 172.21.15.119:0/1765325485 learned_addr learned my addr 172.21.15.119:0/1765325485 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8ffff700 1 -- 172.21.15.119:0/1765325485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8400b200 con 0x7f8e900703f0 2024-01-21T19:59:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8ffff700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e9010aa50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8e8400e040 tx=0x7f8e84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e8400d9f0 con 0x7f8e900703f0 2024-01-21T19:59:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e8401a9f0 con 0x7f8e900703f0 2024-01-21T19:59:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9010b140 con 0x7f8e900703f0 2024-01-21T19:59:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e8401b8c0 con 0x7f8e900703f0 2024-01-21T19:59:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.032+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9010b550 con 0x7f8e900703f0 2024-01-21T19:59:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.033+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f8e8400f070 con 0x7f8e900703f0 2024-01-21T19:59:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.033+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e90045ad0 con 0x7f8e900703f0 2024-01-21T19:59:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.033+0000 7f8e8cff9700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f8e7c02fd70 0x7f8e7c032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.033+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8e840412f0 con 0x7f8e900703f0 2024-01-21T19:59:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.033+0000 7f8e8f7fe700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f8e7c02fd70 0x7f8e7c032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.034+0000 7f8e8f7fe700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f8e7c02fd70 0x7f8e7c032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8e800096a0 tx=0x7f8e80006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.038+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e84022d50 con 0x7f8e900703f0 2024-01-21T19:59:49.207 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.215+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 --> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e9010be30 con 0x7f8e7c02fd70 2024-01-21T19:59:49.210 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.218+0000 7f8e8cff9700 1 -- 172.21.15.119:0/1765325485 <== mgr.14142 v2:172.21.15.119:6800/2754458491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8e9010be30 con 0x7f8e7c02fd70 2024-01-21T19:59:49.210 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f8e7c02fd70 msgr2=0x7f8e7c032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f8e7c02fd70 0x7f8e7c032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8e800096a0 tx=0x7f8e80006b40).stop 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 msgr2=0x7f8e9010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e9010aa50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f8e8400e040 tx=0x7f8e84006ee0).stop 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 shutdown_connections 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f8e7c02fd70 0x7f8e7c032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 --2- 172.21.15.119:0/1765325485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e900703f0 0x7f8e9010aa50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 >> 172.21.15.119:0/1765325485 conn(0x7f8e900f63c0 msgr2=0x7f8e900f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 shutdown_connections 2024-01-21T19:59:49.213 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.220+0000 7f8e965d4700 1 -- 172.21.15.119:0/1765325485 wait complete. 2024-01-21T19:59:49.485 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:48.388330+0000 mon.smithi119 (mon.0) 75 : cluster [DBG] mgrmap e10: smithi119.rqacbm(active, since 1.05981s) 2024-01-21T19:59:49.485 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:48.389414+0000 mgr.smithi119.rqacbm (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-21T19:59:49.485 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:48.391421+0000 mon.smithi119 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:49.485 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:49.220120+0000 mon.smithi119 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:49.574 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying node-exporter service with default placement... 2024-01-21T19:59:49.574 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-21T19:59:49.920 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f576dc08700 1 Processor -- start 2024-01-21T19:59:49.920 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f576dc08700 1 -- start start 2024-01-21T19:59:49.920 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f576dc08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:49.920 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f576dc08700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680fede0 con 0x7f57680704a0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f57677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f57677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44480/0 (socket says 172.21.15.119:44480) 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f57677fe700 1 -- 172.21.15.119:0/731400847 learned_addr learned my addr 172.21.15.119:0/731400847 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f57677fe700 1 -- 172.21.15.119:0/731400847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680fef20 con 0x7f57680704a0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.925+0000 7f57677fe700 1 --2- 172.21.15.119:0/731400847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fe8f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f575800b5f0 tx=0x7f5758006ee0).ready entity=mon.0 client_cookie=82f41bdcc4d4351b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f57667fc700 1 -- 172.21.15.119:0/731400847 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575800d600 con 0x7f57680704a0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f57667fc700 1 -- 172.21.15.119:0/731400847 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f575800dbd0 con 0x7f57680704a0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f57667fc700 1 -- 172.21.15.119:0/731400847 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5758011c00 con 0x7f57680704a0 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 -- 172.21.15.119:0/731400847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 msgr2=0x7f57680fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 --2- 172.21.15.119:0/731400847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fe8f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f575800b5f0 tx=0x7f5758006ee0).stop 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 -- 172.21.15.119:0/731400847 shutdown_connections 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 --2- 172.21.15.119:0/731400847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fe8f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 -- 172.21.15.119:0/731400847 >> 172.21.15.119:0/731400847 conn(0x7f57680f63b0 msgr2=0x7f57680f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:49.921 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 -- 172.21.15.119:0/731400847 shutdown_connections 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.926+0000 7f576dc08700 1 -- 172.21.15.119:0/731400847 wait complete. 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f576dc08700 1 Processor -- start 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f576dc08700 1 -- start start 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f576dc08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f576dc08700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680fa7f0 con 0x7f57680704a0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f57677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f57677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44488/0 (socket says 172.21.15.119:44488) 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.927+0000 7f57677fe700 1 -- 172.21.15.119:0/1923178724 learned_addr learned my addr 172.21.15.119:0/1923178724 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f57677fe700 1 -- 172.21.15.119:0/1923178724 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575800b200 con 0x7f57680704a0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f57677fe700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fc370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f575800b190 tx=0x7f5758006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575801b410 con 0x7f57680704a0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f575801b9e0 con 0x7f57680704a0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575801a880 con 0x7f57680704a0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57680fa990 con 0x7f57680704a0 2024-01-21T19:59:49.922 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.928+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57680fada0 con 0x7f57680704a0 2024-01-21T19:59:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.929+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f575800f070 con 0x7f57680704a0 2024-01-21T19:59:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.929+0000 7f574ffff700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f575402fcf0 0x7f57540321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.929+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5758041320 con 0x7f57680704a0 2024-01-21T19:59:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.929+0000 7f5766ffd700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f575402fcf0 0x7f57540321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.929+0000 7f5766ffd700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f575402fcf0 0x7f57540321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f575c0096a0 tx=0x7f575c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.929+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5768059100 con 0x7f57680704a0 2024-01-21T19:59:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:49.933+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57580226c0 con 0x7f57680704a0 2024-01-21T19:59:50.101 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.109+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 --> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f57680ffc10 con 0x7f575402fcf0 2024-01-21T19:59:50.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.113+0000 7f574ffff700 1 -- 172.21.15.119:0/1923178724 <== mgr.14142 v2:172.21.15.119:6800/2754458491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f57680ffc10 con 0x7f575402fcf0 2024-01-21T19:59:50.105 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f575402fcf0 msgr2=0x7f57540321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f575402fcf0 0x7f57540321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f575c0096a0 tx=0x7f575c006b40).stop 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 msgr2=0x7f57680fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fc370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f575800b190 tx=0x7f5758006ee0).stop 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 shutdown_connections 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f575402fcf0 0x7f57540321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 --2- 172.21.15.119:0/1923178724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f57680704a0 0x7f57680fc370 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 >> 172.21.15.119:0/1923178724 conn(0x7f57680f63b0 msgr2=0x7f57680f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 shutdown_connections 2024-01-21T19:59:50.108 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.115+0000 7f576dc08700 1 -- 172.21.15.119:0/1923178724 wait complete. 2024-01-21T19:59:50.441 INFO:teuthology.orchestra.run.smithi119.stderr:Deploying alertmanager service with default placement... 2024-01-21T19:59:50.441 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf9062700 1 Processor -- start 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf9062700 1 -- start start 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf9062700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf9062700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf40fb490 con 0x7fbaf40fab80 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44500/0 (socket says 172.21.15.119:44500) 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf2d9d700 1 -- 172.21.15.119:0/2305801572 learned_addr learned my addr 172.21.15.119:0/2305801572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.795+0000 7fbaf2d9d700 1 -- 172.21.15.119:0/2305801572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf40fb5d0 con 0x7fbaf40fab80 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf2d9d700 1 --2- 172.21.15.119:0/2305801572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf40fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbae800b870 tx=0x7fbae8006f70).ready entity=mon.0 client_cookie=3d06275fc9ee43a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf1d9b700 1 -- 172.21.15.119:0/2305801572 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae800da20 con 0x7fbaf40fab80 2024-01-21T19:59:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf1d9b700 1 -- 172.21.15.119:0/2305801572 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbae8010470 con 0x7fbaf40fab80 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf1d9b700 1 -- 172.21.15.119:0/2305801572 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae800f5b0 con 0x7fbaf40fab80 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf9062700 1 -- 172.21.15.119:0/2305801572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 msgr2=0x7fbaf40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf9062700 1 --2- 172.21.15.119:0/2305801572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf40fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fbae800b870 tx=0x7fbae8006f70).stop 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf9062700 1 -- 172.21.15.119:0/2305801572 shutdown_connections 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf9062700 1 --2- 172.21.15.119:0/2305801572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf40fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf9062700 1 -- 172.21.15.119:0/2305801572 >> 172.21.15.119:0/2305801572 conn(0x7fbaf40f63c0 msgr2=0x7fbaf40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.796+0000 7fbaf9062700 1 -- 172.21.15.119:0/2305801572 shutdown_connections 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.797+0000 7fbaf9062700 1 -- 172.21.15.119:0/2305801572 wait complete. 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.797+0000 7fbaf9062700 1 Processor -- start 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.797+0000 7fbaf9062700 1 -- start start 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.797+0000 7fbaf9062700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf410cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.797+0000 7fbaf9062700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf410d070 con 0x7fbaf40fab80 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf410cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf410cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44502/0 (socket says 172.21.15.119:44502) 2024-01-21T19:59:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf2d9d700 1 -- 172.21.15.119:0/2311413279 learned_addr learned my addr 172.21.15.119:0/2311413279 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf2d9d700 1 -- 172.21.15.119:0/2311413279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae800b480 con 0x7fbaf40fab80 2024-01-21T19:59:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf2d9d700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf410cb80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbae8006800 tx=0x7fbae8006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae800f980 con 0x7fbaf40fab80 2024-01-21T19:59:50.792 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbae8019450 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf410d270 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae8023640 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.798+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf410d680 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.799+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbae8016030 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.799+0000 7fbadb7fe700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fbae002feb0 0x7fbae0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.799+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbae8041f40 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.799+0000 7fbaf259c700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fbae002feb0 0x7fbae0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.799+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf4059100 con 0x7fbaf40fab80 2024-01-21T19:59:50.793 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.799+0000 7fbaf259c700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fbae002feb0 0x7fbae0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbae40096a0 tx=0x7fbae4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:50.795 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.803+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbae8014080 con 0x7fbaf40fab80 2024-01-21T19:59:50.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:49.217365+0000 mgr.smithi119.rqacbm (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:50.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:49.217895+0000 mgr.smithi119.rqacbm (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-21T19:59:50.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:50.114599+0000 mon.smithi119 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:50.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:50.224458+0000 mon.smithi119 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi119.rqacbm(active, since 2s) 2024-01-21T19:59:50.971 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.979+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 --> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaf41026b0 con 0x7fbae002feb0 2024-01-21T19:59:50.974 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.982+0000 7fbadb7fe700 1 -- 172.21.15.119:0/2311413279 <== mgr.14142 v2:172.21.15.119:6800/2754458491 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fbaf41026b0 con 0x7fbae002feb0 2024-01-21T19:59:50.974 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fbae002feb0 msgr2=0x7fbae0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fbae002feb0 0x7fbae0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbae40096a0 tx=0x7fbae4006b40).stop 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 msgr2=0x7fbaf410cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf410cb80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fbae8006800 tx=0x7fbae8006f70).stop 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 shutdown_connections 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fbae002feb0 0x7fbae0032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 --2- 172.21.15.119:0/2311413279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbaf40fab80 0x7fbaf410cb80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 >> 172.21.15.119:0/2311413279 conn(0x7fbaf40f63c0 msgr2=0x7fbaf40ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 shutdown_connections 2024-01-21T19:59:50.977 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:50.984+0000 7fbaf9062700 1 -- 172.21.15.119:0/2311413279 wait complete. 2024-01-21T19:59:51.332 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-21T19:59:51.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:50.111796+0000 mgr.smithi119.rqacbm (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:51.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:50.112454+0000 mgr.smithi119.rqacbm (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-21T19:59:51.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:50.983998+0000 mon.smithi119 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.119:0/4226629128' entity='mgr.smithi119.rqacbm' 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.660+0000 7fe804f16700 1 Processor -- start 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.660+0000 7fe804f16700 1 -- start start 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe804f16700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe804f16700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000fdc50 con 0x7fe8000fd340 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fe59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fe59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44510/0 (socket says 172.21.15.119:44510) 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fe59c700 1 -- 172.21.15.119:0/3902693295 learned_addr learned my addr 172.21.15.119:0/3902693295 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fe59c700 1 -- 172.21.15.119:0/3902693295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8000fdd90 con 0x7fe8000fd340 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fe59c700 1 --2- 172.21.15.119:0/3902693295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fd760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe7e800b5f0 tx=0x7fe7e8006ee0).ready entity=mon.0 client_cookie=b970b30146304e57 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fd59a700 1 -- 172.21.15.119:0/3902693295 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e800d600 con 0x7fe8000fd340 2024-01-21T19:59:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fd59a700 1 -- 172.21.15.119:0/3902693295 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe7e800dbd0 con 0x7fe8000fd340 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.661+0000 7fe7fd59a700 1 -- 172.21.15.119:0/3902693295 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e8011c00 con 0x7fe8000fd340 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 -- 172.21.15.119:0/3902693295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 msgr2=0x7fe8000fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 --2- 172.21.15.119:0/3902693295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fd760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fe7e800b5f0 tx=0x7fe7e8006ee0).stop 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 -- 172.21.15.119:0/3902693295 shutdown_connections 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 --2- 172.21.15.119:0/3902693295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fd760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 -- 172.21.15.119:0/3902693295 >> 172.21.15.119:0/3902693295 conn(0x7fe8000f5b40 msgr2=0x7fe8000f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 -- 172.21.15.119:0/3902693295 shutdown_connections 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 -- 172.21.15.119:0/3902693295 wait complete. 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.662+0000 7fe804f16700 1 Processor -- start 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe804f16700 1 -- start start 2024-01-21T19:59:51.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe804f16700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:51.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe804f16700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8000fb090 con 0x7fe8000fd340 2024-01-21T19:59:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7fe59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7fe59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fcc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44514/0 (socket says 172.21.15.119:44514) 2024-01-21T19:59:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7fe59c700 1 -- 172.21.15.119:0/1233621058 learned_addr learned my addr 172.21.15.119:0/1233621058 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7fe59c700 1 -- 172.21.15.119:0/1233621058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e800b200 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7fe59c700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fcc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe7e8006c70 tx=0x7fe7e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e800d9f0 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8000fb230 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe7e801a9f0 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e801b890 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.663+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8000fb640 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.664+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe7e800f070 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.664+0000 7fe7f6ffd700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe7e002fe60 0x7fe7e0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.664+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe7e8041450 con 0x7fe8000fd340 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.664+0000 7fe7fdd9b700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe7e002fe60 0x7fe7e0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.665+0000 7fe7fdd9b700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe7e002fe60 0x7fe7e0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe7f00096a0 tx=0x7fe7f0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.665+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe800059100 con 0x7fe8000fd340 2024-01-21T19:59:51.661 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.670+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7e8025460 con 0x7fe8000fd340 2024-01-21T19:59:51.809 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.817+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe8000fa6b0 con 0x7fe8000fd340 2024-01-21T19:59:51.812 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.820+0000 7fe7f6ffd700 1 -- 172.21.15.119:0/1233621058 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe7e8006a70 con 0x7fe8000fd340 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe7e002fe60 msgr2=0x7fe7e0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe7e002fe60 0x7fe7e0032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe7f00096a0 tx=0x7fe7f0006b40).stop 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 msgr2=0x7fe8000fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fcc10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe7e8006c70 tx=0x7fe7e8006ee0).stop 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 shutdown_connections 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fe7e002fe60 0x7fe7e0032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 --2- 172.21.15.119:0/1233621058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8000fd340 0x7fe8000fcc10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 >> 172.21.15.119:0/1233621058 conn(0x7fe8000f5b40 msgr2=0x7fe8000ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 shutdown_connections 2024-01-21T19:59:51.815 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:51.822+0000 7fe804f16700 1 -- 172.21.15.119:0/1233621058 wait complete. 2024-01-21T19:59:52.149 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.495+0000 7fed38a71700 1 Processor -- start 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.495+0000 7fed38a71700 1 -- start start 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed38a71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed340fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed38a71700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340fb490 con 0x7fed340fab80 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed340fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed340fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41370/0 (socket says 172.21.15.119:41370) 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3259c700 1 -- 172.21.15.119:0/433503987 learned_addr learned my addr 172.21.15.119:0/433503987 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3259c700 1 -- 172.21.15.119:0/433503987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed340fb5d0 con 0x7fed340fab80 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3259c700 1 --2- 172.21.15.119:0/433503987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed340fafa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fed1c00b870 tx=0x7fed1c006f70).ready entity=mon.0 client_cookie=1f4d251c864af325 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3159a700 1 -- 172.21.15.119:0/433503987 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed1c00da20 con 0x7fed340fab80 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3159a700 1 -- 172.21.15.119:0/433503987 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fed1c010470 con 0x7fed340fab80 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.496+0000 7fed3159a700 1 -- 172.21.15.119:0/433503987 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed1c00f5b0 con 0x7fed340fab80 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 -- 172.21.15.119:0/433503987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 msgr2=0x7fed340fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 --2- 172.21.15.119:0/433503987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed340fafa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fed1c00b870 tx=0x7fed1c006f70).stop 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 -- 172.21.15.119:0/433503987 shutdown_connections 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 --2- 172.21.15.119:0/433503987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed340fafa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 -- 172.21.15.119:0/433503987 >> 172.21.15.119:0/433503987 conn(0x7fed340f63c0 msgr2=0x7fed340f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 -- 172.21.15.119:0/433503987 shutdown_connections 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.497+0000 7fed38a71700 1 -- 172.21.15.119:0/433503987 wait complete. 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed38a71700 1 Processor -- start 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed38a71700 1 -- start start 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed38a71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed34108780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed38a71700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34108c70 con 0x7fed340fab80 2024-01-21T19:59:52.491 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed3259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed34108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:52.492 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed3259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed34108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41376/0 (socket says 172.21.15.119:41376) 2024-01-21T19:59:52.492 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed3259c700 1 -- 172.21.15.119:0/4277658331 learned_addr learned my addr 172.21.15.119:0/4277658331 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:52.492 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed3259c700 1 -- 172.21.15.119:0/4277658331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c00b480 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed3259c700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed34108780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fed1c006a40 tx=0x7fed1c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed1c00f3d0 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fed1c00fef0 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed34108e70 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.498+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed1c019d40 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.499+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed34109280 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.499+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fed1c02b060 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.499+0000 7fed2affd700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fed1402fe60 0x7fed14032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.499+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fed1c044250 con 0x7fed340fab80 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.499+0000 7fed31d9b700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fed1402fe60 0x7fed14032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.500+0000 7fed31d9b700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fed1402fe60 0x7fed14032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fed240096a0 tx=0x7fed24006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:52.493 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.500+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed34059100 con 0x7fed340fab80 2024-01-21T19:59:52.496 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.504+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed1c014070 con 0x7fed340fab80 2024-01-21T19:59:52.645 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.653+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fed34102640 con 0x7fed340fab80 2024-01-21T19:59:52.650 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.659+0000 7fed2affd700 1 -- 172.21.15.119:0/4277658331 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fed1c018ba0 con 0x7fed340fab80 2024-01-21T19:59:52.653 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fed1402fe60 msgr2=0x7fed14032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fed1402fe60 0x7fed14032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fed240096a0 tx=0x7fed24006b40).stop 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 msgr2=0x7fed34108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed34108780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fed1c006a40 tx=0x7fed1c006f70).stop 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 shutdown_connections 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fed1402fe60 0x7fed14032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 --2- 172.21.15.119:0/4277658331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed340fab80 0x7fed34108780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 >> 172.21.15.119:0/4277658331 conn(0x7fed340f63c0 msgr2=0x7fed340ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 shutdown_connections 2024-01-21T19:59:52.654 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:52.661+0000 7fed38a71700 1 -- 172.21.15.119:0/4277658331 wait complete. 2024-01-21T19:59:52.672 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:50.981247+0000 mgr.smithi119.rqacbm (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T19:59:52.672 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T19:59:50.981742+0000 mgr.smithi119.rqacbm (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-21T19:59:52.672 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:51.822039+0000 mon.smithi119 (mon.0) 81 : audit [INF] from='client.? 172.21.15.119:0/1233621058' entity='client.admin' 2024-01-21T19:59:53.007 INFO:teuthology.orchestra.run.smithi119.stderr:Enabling the dashboard module... 2024-01-21T19:59:53.007 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-21T19:59:53.327 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc27e47a700 1 Processor -- start 2024-01-21T19:59:53.327 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc27e47a700 1 -- start start 2024-01-21T19:59:53.327 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc27e47a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:53.327 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc27e47a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780fedd0 con 0x7fc278070430 2024-01-21T19:59:53.327 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc277fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:53.327 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc277fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41378/0 (socket says 172.21.15.119:41378) 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc277fff700 1 -- 172.21.15.119:0/1103607278 learned_addr learned my addr 172.21.15.119:0/1103607278 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.333+0000 7fc277fff700 1 -- 172.21.15.119:0/1103607278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2780fef10 con 0x7fc278070430 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc277fff700 1 --2- 172.21.15.119:0/1103607278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc26800b5f0 tx=0x7fc268006ee0).ready entity=mon.0 client_cookie=491911977540eddf server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc276ffd700 1 -- 172.21.15.119:0/1103607278 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26800d600 con 0x7fc278070430 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc276ffd700 1 -- 172.21.15.119:0/1103607278 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc26800dbd0 con 0x7fc278070430 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc276ffd700 1 -- 172.21.15.119:0/1103607278 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc268011c00 con 0x7fc278070430 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 -- 172.21.15.119:0/1103607278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 msgr2=0x7fc2780fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 --2- 172.21.15.119:0/1103607278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc26800b5f0 tx=0x7fc268006ee0).stop 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 -- 172.21.15.119:0/1103607278 shutdown_connections 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 --2- 172.21.15.119:0/1103607278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fe8e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 -- 172.21.15.119:0/1103607278 >> 172.21.15.119:0/1103607278 conn(0x7fc2780f63c0 msgr2=0x7fc2780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:53.328 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 -- 172.21.15.119:0/1103607278 shutdown_connections 2024-01-21T19:59:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.334+0000 7fc27e47a700 1 -- 172.21.15.119:0/1103607278 wait complete. 2024-01-21T19:59:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc27e47a700 1 Processor -- start 2024-01-21T19:59:53.329 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc27e47a700 1 -- start start 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc27e47a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc27e47a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780fa7e0 con 0x7fc278070430 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc277fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc277fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41390/0 (socket says 172.21.15.119:41390) 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.335+0000 7fc277fff700 1 -- 172.21.15.119:0/2393087667 learned_addr learned my addr 172.21.15.119:0/2393087667 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc277fff700 1 -- 172.21.15.119:0/2393087667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26800b200 con 0x7fc278070430 2024-01-21T19:59:53.330 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc277fff700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fc360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc26800b190 tx=0x7fc268006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26801b410 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc26801b9e0 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc26801a880 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2780fa980 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.336+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2780fad90 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.337+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc26800f070 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.337+0000 7fc274ff9700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fc26402fe60 0x7fc264032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.337+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc268041510 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.337+0000 7fc2777fe700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fc26402fe60 0x7fc264032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.337+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc278059100 con 0x7fc278070430 2024-01-21T19:59:53.331 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.337+0000 7fc2777fe700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fc26402fe60 0x7fc264032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc26c0096a0 tx=0x7fc26c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.341+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc268016070 con 0x7fc278070430 2024-01-21T19:59:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.529+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc278045ad0 con 0x7fc278070430 2024-01-21T19:59:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.659+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc268025980 con 0x7fc278070430 2024-01-21T19:59:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.660+0000 7fc274ff9700 1 -- 172.21.15.119:0/2393087667 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc268010360 con 0x7fc278070430 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fc26402fe60 msgr2=0x7fc264032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fc26402fe60 0x7fc264032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc26c0096a0 tx=0x7fc26c006b40).stop 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 msgr2=0x7fc2780fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fc360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fc26800b190 tx=0x7fc268006ee0).stop 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 shutdown_connections 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7fc26402fe60 0x7fc264032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 --2- 172.21.15.119:0/2393087667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278070430 0x7fc2780fc360 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 >> 172.21.15.119:0/2393087667 conn(0x7fc2780f63c0 msgr2=0x7fc2780f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 shutdown_connections 2024-01-21T19:59:53.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:53.662+0000 7fc27e47a700 1 -- 172.21.15.119:0/2393087667 wait complete. 2024-01-21T19:59:53.934 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:52.660639+0000 mon.smithi119 (mon.0) 82 : audit [INF] from='client.? 172.21.15.119:0/4277658331' entity='client.admin' 2024-01-21T19:59:53.934 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:53.531304+0000 mon.smithi119 (mon.0) 83 : audit [INF] from='client.? 172.21.15.119:0/2393087667' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-21T19:59:53.990 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f53a19da700 1 Processor -- start 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f53a19da700 1 -- start start 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f53a19da700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c100d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f53a19da700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0fc760 con 0x7f539c070420 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f539affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f539affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c100d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41418/0 (socket says 172.21.15.119:41418) 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f539affd700 1 -- 172.21.15.119:0/2697216162 learned_addr learned my addr 172.21.15.119:0/2697216162 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.354+0000 7f539affd700 1 -- 172.21.15.119:0/2697216162 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c101260 con 0x7f539c070420 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f539affd700 1 --2- 172.21.15.119:0/2697216162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c100d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f538c00b5f0 tx=0x7f538c006ee0).ready entity=mon.0 client_cookie=76aa1332d4fe67b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f5399ffb700 1 -- 172.21.15.119:0/2697216162 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538c00d600 con 0x7f539c070420 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f5399ffb700 1 -- 172.21.15.119:0/2697216162 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f538c00dbd0 con 0x7f539c070420 2024-01-21T19:59:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f5399ffb700 1 -- 172.21.15.119:0/2697216162 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538c011c00 con 0x7f539c070420 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 -- 172.21.15.119:0/2697216162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 msgr2=0x7f539c100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 --2- 172.21.15.119:0/2697216162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c100d70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f538c00b5f0 tx=0x7f538c006ee0).stop 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 -- 172.21.15.119:0/2697216162 shutdown_connections 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 --2- 172.21.15.119:0/2697216162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c100d70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 -- 172.21.15.119:0/2697216162 >> 172.21.15.119:0/2697216162 conn(0x7f539c0f63b0 msgr2=0x7f539c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 -- 172.21.15.119:0/2697216162 shutdown_connections 2024-01-21T19:59:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.355+0000 7f53a19da700 1 -- 172.21.15.119:0/2697216162 wait complete. 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.356+0000 7f53a19da700 1 Processor -- start 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.356+0000 7f53a19da700 1 -- start start 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53a19da700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c10cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53a19da700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c10d0d0 con 0x7f539c070420 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f539affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c10cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f539affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c10cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41420/0 (socket says 172.21.15.119:41420) 2024-01-21T19:59:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f539affd700 1 -- 172.21.15.119:0/3353527034 learned_addr learned my addr 172.21.15.119:0/3353527034 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f539affd700 1 -- 172.21.15.119:0/3353527034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c00b200 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f539affd700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c10cbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f538c00b190 tx=0x7f538c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538c01b410 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f538c01b9e0 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c10d2d0 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538c01a880 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.357+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539c10d6e0 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.358+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f538c00f070 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.358+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539c045ad0 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.358+0000 7f53837fe700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 0x7f5388032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.359+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f538c041470 con 0x7f539c070420 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.359+0000 7f539a7fc700 1 -- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 msgr2=0x7f5388032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.359+0000 7f539a7fc700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 0x7f5388032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:54.354 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.363+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f538c022b90 con 0x7f539c070420 2024-01-21T19:59:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.559+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f539c10e790 con 0x7f539c070420 2024-01-21T19:59:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.559+0000 7f53837fe700 1 -- 172.21.15.119:0/3353527034 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f538c016070 con 0x7f539c070420 2024-01-21T19:59:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.560+0000 7f539a7fc700 1 -- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 msgr2=0x7f5388032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.560+0000 7f539a7fc700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 0x7f5388032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "epoch": 12, 2024-01-21T19:59:54.553 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "available": true, 2024-01-21T19:59:54.553 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "active_name": "smithi119.rqacbm", 2024-01-21T19:59:54.553 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-21T19:59:54.553 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 msgr2=0x7f5388032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 0x7f5388032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 msgr2=0x7f539c10cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c10cbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f538c00b190 tx=0x7f538c006ee0).stop 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 shutdown_connections 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f538802feb0 0x7f5388032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 --2- 172.21.15.119:0/3353527034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f539c070420 0x7f539c10cbe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 >> 172.21.15.119:0/3353527034 conn(0x7f539c0f63b0 msgr2=0x7f539c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:54.554 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 shutdown_connections 2024-01-21T19:59:54.555 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:54.561+0000 7f53a19da700 1 -- 172.21.15.119:0/3353527034 wait complete. 2024-01-21T19:59:54.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:53.661947+0000 mon.smithi119 (mon.0) 84 : audit [INF] from='client.? 172.21.15.119:0/2393087667' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-21T19:59:54.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:53.661994+0000 mon.smithi119 (mon.0) 85 : cluster [DBG] mgrmap e12: smithi119.rqacbm(active, since 6s) 2024-01-21T19:59:54.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:54.561300+0000 mon.smithi119 (mon.0) 86 : audit [DBG] from='client.? 172.21.15.119:0/3353527034' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-21T19:59:54.950 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for the mgr to restart... 2024-01-21T19:59:54.950 INFO:teuthology.orchestra.run.smithi119.stderr:Waiting for mgr epoch 12... 2024-01-21T19:59:54.950 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-21T19:59:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.303+0000 7f946b65e700 1 Processor -- start 2024-01-21T19:59:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.303+0000 7f946b65e700 1 -- start start 2024-01-21T19:59:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f946b65e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f94640fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f946b65e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94640fb490 con 0x7f94640fab80 2024-01-21T19:59:55.296 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f94693fa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f94640fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:55.298 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f94693fa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f94640fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41426/0 (socket says 172.21.15.119:41426) 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f94693fa700 1 -- 172.21.15.119:0/1470332446 learned_addr learned my addr 172.21.15.119:0/1470332446 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f94693fa700 1 -- 172.21.15.119:0/1470332446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94640fb5d0 con 0x7f94640fab80 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f94693fa700 1 --2- 172.21.15.119:0/1470332446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f94640fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f946000b870 tx=0x7f9460006f70).ready entity=mon.0 client_cookie=8ccec2c481713647 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f9453fff700 1 -- 172.21.15.119:0/1470332446 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f946000da20 con 0x7f94640fab80 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f9453fff700 1 -- 172.21.15.119:0/1470332446 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9460010470 con 0x7f94640fab80 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.304+0000 7f9453fff700 1 -- 172.21.15.119:0/1470332446 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f946000f5b0 con 0x7f94640fab80 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 -- 172.21.15.119:0/1470332446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 msgr2=0x7f94640fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 --2- 172.21.15.119:0/1470332446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f94640fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f946000b870 tx=0x7f9460006f70).stop 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 -- 172.21.15.119:0/1470332446 shutdown_connections 2024-01-21T19:59:55.299 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 --2- 172.21.15.119:0/1470332446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f94640fafa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:55.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 -- 172.21.15.119:0/1470332446 >> 172.21.15.119:0/1470332446 conn(0x7f94640f63a0 msgr2=0x7f94640f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:55.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 -- 172.21.15.119:0/1470332446 shutdown_connections 2024-01-21T19:59:55.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.305+0000 7f946b65e700 1 -- 172.21.15.119:0/1470332446 wait complete. 2024-01-21T19:59:55.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f946b65e700 1 Processor -- start 2024-01-21T19:59:55.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f946b65e700 1 -- start start 2024-01-21T19:59:55.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f946b65e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f9464108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:55.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f946b65e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9464108c10 con 0x7f94640fab80 2024-01-21T19:59:55.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f94693fa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f9464108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:55.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f94693fa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f9464108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41428/0 (socket says 172.21.15.119:41428) 2024-01-21T19:59:55.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.306+0000 7f94693fa700 1 -- 172.21.15.119:0/2180054072 learned_addr learned my addr 172.21.15.119:0/2180054072 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T19:59:55.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f94693fa700 1 -- 172.21.15.119:0/2180054072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946000b480 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f94693fa700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f9464108720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f9460006a40 tx=0x7f9460006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f946000f3d0 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f946000fef0 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9464108e10 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9460019d40 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.307+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9464109220 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.308+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f946002b060 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.308+0000 7f9451ffb700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.308+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 --> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c032a30 con 0x7f944c02feb0 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.308+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9460044340 con 0x7f94640fab80 2024-01-21T19:59:55.302 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.308+0000 7f9468bf9700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 msgr2=0x7f944c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:55.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.308+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-21T19:59:55.501 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.509+0000 7f9468bf9700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 msgr2=0x7f944c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:55.501 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.509+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-21T19:59:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.910+0000 7f9468bf9700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 msgr2=0x7f944c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:55.910+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-21T19:59:56.703 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:56.711+0000 7f9468bf9700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 msgr2=0x7f944c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:56.703 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:56.711+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-21T19:59:58.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:58.312+0000 7f9468bf9700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 msgr2=0x7f944c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/2754458491 2024-01-21T19:59:58.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:58.312+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-21T19:59:58.637 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:58.645+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f9460016030 con 0x7f94640fab80 2024-01-21T19:59:58.637 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:58.645+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 msgr2=0x7f944c032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-21T19:59:58.637 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:58.645+0000 7f9451ffb700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:59.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:58.594306+0000 mon.smithi119 (mon.0) 87 : cluster [INF] Active manager daemon smithi119.rqacbm restarted 2024-01-21T19:59:59.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:58.594700+0000 mon.smithi119 (mon.0) 88 : cluster [INF] Activating manager daemon smithi119.rqacbm 2024-01-21T19:59:59.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:58.596469+0000 mon.smithi119 (mon.0) 89 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-21T19:59:59.649 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.656+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f94600194e0 con 0x7f94640fab80 2024-01-21T19:59:59.649 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.656+0000 7f9451ffb700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f944c033b40 0x7f944c035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T19:59:59.649 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.656+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f944c032a30 con 0x7f944c033b40 2024-01-21T19:59:59.649 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.656+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f944c033b40 0x7f944c035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T19:59:59.649 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.656+0000 7f9468bf9700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f944c033b40 0x7f944c035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9454006fd0 tx=0x7f9454006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T19:59:59.649 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.657+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mgr.14162 v2:172.21.15.119:6800/1043201922 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f944c032a30 con 0x7f944c033b40 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.663+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9464059100 con 0x7f944c033b40 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.663+0000 7f9451ffb700 1 -- 172.21.15.119:0/2180054072 <== mgr.14162 v2:172.21.15.119:6800/1043201922 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f9464059100 con 0x7f944c033b40 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: { 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: "initialized": true 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: } 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f944c033b40 msgr2=0x7f944c035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f944c033b40 0x7f944c035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f9454006fd0 tx=0x7f9454006cf0).stop 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 msgr2=0x7f9464108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f9464108720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f9460006a40 tx=0x7f9460006f70).stop 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 shutdown_connections 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f944c033b40 0x7f944c035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:59.657 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94640fab80 0x7f9464108720 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T19:59:59.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 --2- 172.21.15.119:0/2180054072 >> [v2:172.21.15.119:6800/2754458491,v1:172.21.15.119:6801/2754458491] conn(0x7f944c02feb0 0x7f944c032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-21T19:59:59.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 >> 172.21.15.119:0/2180054072 conn(0x7f94640f63a0 msgr2=0x7f94640ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T19:59:59.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 shutdown_connections 2024-01-21T19:59:59.658 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T19:59:59.664+0000 7f946b65e700 1 -- 172.21.15.119:0/2180054072 wait complete. 2024-01-21T19:59:59.966 INFO:teuthology.orchestra.run.smithi119.stderr:mgr epoch 12 is available 2024-01-21T19:59:59.966 INFO:teuthology.orchestra.run.smithi119.stderr:Generating a dashboard self-signed certificate... 2024-01-21T19:59:59.966 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:58.647450+0000 mon.smithi119 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi119.rqacbm(active, starting, since 0.0528361s) 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.648604+0000 mon.smithi119 (mon.0) 91 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata", "id": "smithi119"}]: dispatch 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.649195+0000 mon.smithi119 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mgr metadata", "who": "smithi119.rqacbm", "id": "smithi119.rqacbm"}]: dispatch 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.649821+0000 mon.smithi119 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.650025+0000 mon.smithi119 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.650217+0000 mon.smithi119 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-21T19:59:59.967 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:58.663854+0000 mon.smithi119 (mon.0) 96 : cluster [INF] Manager daemon smithi119.rqacbm is now available 2024-01-21T19:59:59.968 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.759843+0000 mon.smithi119 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:59.968 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.761531+0000 mon.smithi119 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-21T19:59:59.968 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.777932+0000 mon.smithi119 (mon.0) 99 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T19:59:59.968 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 19:59:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:58.780226+0000 mon.smithi119 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:00:00.300 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.307+0000 7f69f629a700 1 Processor -- start 2024-01-21T20:00:00.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.307+0000 7f69f629a700 1 -- start start 2024-01-21T20:00:00.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.307+0000 7f69f629a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:00.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.307+0000 7f69f629a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00fec60 con 0x7f69f00703f0 2024-01-21T20:00:00.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69f5298700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:00.301 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69f5298700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41484/0 (socket says 172.21.15.119:41484) 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69f5298700 1 -- 172.21.15.119:0/2477685472 learned_addr learned my addr 172.21.15.119:0/2477685472 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69f5298700 1 -- 172.21.15.119:0/2477685472 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f00feda0 con 0x7f69f00703f0 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69f5298700 1 --2- 172.21.15.119:0/2477685472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fe770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f69ec00b5f0 tx=0x7f69ec006ee0).ready entity=mon.0 client_cookie=2ae37126c8a659a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69dffff700 1 -- 172.21.15.119:0/2477685472 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69ec00d600 con 0x7f69f00703f0 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69dffff700 1 -- 172.21.15.119:0/2477685472 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f69ec00dbd0 con 0x7f69f00703f0 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.308+0000 7f69dffff700 1 -- 172.21.15.119:0/2477685472 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69ec011c00 con 0x7f69f00703f0 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- 172.21.15.119:0/2477685472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 msgr2=0x7f69f00fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 --2- 172.21.15.119:0/2477685472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fe770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f69ec00b5f0 tx=0x7f69ec006ee0).stop 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- 172.21.15.119:0/2477685472 shutdown_connections 2024-01-21T20:00:00.303 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 --2- 172.21.15.119:0/2477685472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fe770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- 172.21.15.119:0/2477685472 >> 172.21.15.119:0/2477685472 conn(0x7f69f00f6210 msgr2=0x7f69f00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- 172.21.15.119:0/2477685472 shutdown_connections 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- 172.21.15.119:0/2477685472 wait complete. 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 Processor -- start 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- start start 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.309+0000 7f69f629a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00fa670 con 0x7f69f00703f0 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f5298700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f5298700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41500/0 (socket says 172.21.15.119:41500) 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f5298700 1 -- 172.21.15.119:0/3160716444 learned_addr learned my addr 172.21.15.119:0/3160716444 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f5298700 1 -- 172.21.15.119:0/3160716444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec00b200 con 0x7f69f00703f0 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f5298700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fc1f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f69ec00b190 tx=0x7f69ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69ec01b410 con 0x7f69f00703f0 2024-01-21T20:00:00.304 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00fa810 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f69ec01b9e0 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f00fac20 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.310+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69ec01a800 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.311+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f69ec00f070 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.311+0000 7f69ddffb700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69d802fd50 0x7f69d8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.311+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69ec042360 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.311+0000 7f69f4a97700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69d802fd50 0x7f69d8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.312+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f0059100 con 0x7f69f00703f0 2024-01-21T20:00:00.305 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.312+0000 7f69f4a97700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69d802fd50 0x7f69d8032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f69e00096a0 tx=0x7f69e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:00.307 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.316+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69ec016080 con 0x7f69f00703f0 2024-01-21T20:00:00.459 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.467+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f69f0104710 con 0x7f69d802fd50 2024-01-21T20:00:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.508+0000 7f69ddffb700 1 -- 172.21.15.119:0/3160716444 <== mgr.14162 v2:172.21.15.119:6800/1043201922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f69f0104710 con 0x7f69d802fd50 2024-01-21T20:00:00.500 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-21T20:00:00.502 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69d802fd50 msgr2=0x7f69d8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:00.502 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69d802fd50 0x7f69d8032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f69e00096a0 tx=0x7f69e0006b40).stop 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 msgr2=0x7f69f00fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fc1f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f69ec00b190 tx=0x7f69ec006ee0).stop 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 shutdown_connections 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69d802fd50 0x7f69d8032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 --2- 172.21.15.119:0/3160716444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69f00703f0 0x7f69f00fc1f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 >> 172.21.15.119:0/3160716444 conn(0x7f69f00f6210 msgr2=0x7f69f00f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.510+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 shutdown_connections 2024-01-21T20:00:00.503 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:00.511+0000 7f69f629a700 1 -- 172.21.15.119:0/3160716444 wait complete. 2024-01-21T20:00:00.791 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:59.658631+0000 mgr.smithi119.rqacbm (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-21T20:00:00.791 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T19:59:59.659406+0000 mon.smithi119 (mon.0) 101 : cluster [DBG] mgrmap e14: smithi119.rqacbm(active, since 1.06479s) 2024-01-21T20:00:00.791 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T19:59:59.665142+0000 mgr.smithi119.rqacbm (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-21T20:00:00.791 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:00.504671+0000 mon.smithi119 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:00.791 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:00.509384+0000 mon.smithi119 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:00.830 INFO:teuthology.orchestra.run.smithi119.stderr:Creating initial admin user... 2024-01-21T20:00:00.831 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpow74pmw3:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.445+0000 7f86f749e700 1 Processor -- start 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.445+0000 7f86f749e700 1 -- start start 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f749e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f749e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f00fb490 con 0x7f86f00fab80 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f523a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f523a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41504/0 (socket says 172.21.15.119:41504) 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f523a700 1 -- 172.21.15.119:0/4067090128 learned_addr learned my addr 172.21.15.119:0/4067090128 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f523a700 1 -- 172.21.15.119:0/4067090128 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f00fb5d0 con 0x7f86f00fab80 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86f523a700 1 --2- 172.21.15.119:0/4067090128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f00fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f86e400b5f0 tx=0x7f86e4006ee0).ready entity=mon.0 client_cookie=a7d6f79384d6c47d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86e3fff700 1 -- 172.21.15.119:0/4067090128 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e400d600 con 0x7f86f00fab80 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86e3fff700 1 -- 172.21.15.119:0/4067090128 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f86e400dbd0 con 0x7f86f00fab80 2024-01-21T20:00:01.444 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.446+0000 7f86e3fff700 1 -- 172.21.15.119:0/4067090128 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e4011c00 con 0x7f86f00fab80 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.447+0000 7f86f749e700 1 -- 172.21.15.119:0/4067090128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 msgr2=0x7f86f00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.447+0000 7f86f749e700 1 --2- 172.21.15.119:0/4067090128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f00fafa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f86e400b5f0 tx=0x7f86e4006ee0).stop 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.447+0000 7f86f749e700 1 -- 172.21.15.119:0/4067090128 shutdown_connections 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.447+0000 7f86f749e700 1 --2- 172.21.15.119:0/4067090128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f00fafa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.447+0000 7f86f749e700 1 -- 172.21.15.119:0/4067090128 >> 172.21.15.119:0/4067090128 conn(0x7f86f00f63c0 msgr2=0x7f86f00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.448+0000 7f86f749e700 1 -- 172.21.15.119:0/4067090128 shutdown_connections 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.448+0000 7f86f749e700 1 -- 172.21.15.119:0/4067090128 wait complete. 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.448+0000 7f86f749e700 1 Processor -- start 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.448+0000 7f86f749e700 1 -- start start 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f749e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f010d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f749e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f0070470 con 0x7f86f00fab80 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f523a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f010d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f523a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f010d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41516/0 (socket says 172.21.15.119:41516) 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f523a700 1 -- 172.21.15.119:0/2688693312 learned_addr learned my addr 172.21.15.119:0/2688693312 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:01.445 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f523a700 1 -- 172.21.15.119:0/2688693312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e400b200 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f523a700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f010d750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f86e400e040 tx=0x7f86e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e401b410 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f86e401b9e0 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f0070670 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86e401a820 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.449+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f010de00 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.450+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f86e400f070 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.450+0000 7f86e1ffb700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f86d802fd50 0x7f86d8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.450+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86e40414b0 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.450+0000 7f86f4a39700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f86d802fd50 0x7f86d8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.450+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f0045ad0 con 0x7f86f00fab80 2024-01-21T20:00:01.446 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.451+0000 7f86f4a39700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f86d802fd50 0x7f86d8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f86ec0096a0 tx=0x7f86ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:01.447 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.455+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86e4016070 con 0x7f86f00fab80 2024-01-21T20:00:01.501 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.509+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f86e40257e0 con 0x7f86f00fab80 2024-01-21T20:00:01.612 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.620+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f86f0058f20 con 0x7f86d802fd50 2024-01-21T20:00:01.828 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.836+0000 7f86e1ffb700 1 -- 172.21.15.119:0/2688693312 <== mgr.14162 v2:172.21.15.119:6800/1043201922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f86f0058f20 con 0x7f86d802fd50 2024-01-21T20:00:01.828 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$6xFU8cMFMbURT4777oSPG.zpVLN.jMZCIMZS6ZtQ3Af0JwnQAkr/K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705867201, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f86d802fd50 msgr2=0x7f86d8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f86d802fd50 0x7f86d8032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f86ec0096a0 tx=0x7f86ec006b40).stop 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 msgr2=0x7f86f010d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f010d750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f86e400e040 tx=0x7f86e4006ee0).stop 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 shutdown_connections 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f86d802fd50 0x7f86d8032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 --2- 172.21.15.119:0/2688693312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86f00fab80 0x7f86f010d750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 >> 172.21.15.119:0/2688693312 conn(0x7f86f00f63c0 msgr2=0x7f86f00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 shutdown_connections 2024-01-21T20:00:01.831 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:01.838+0000 7f86f749e700 1 -- 172.21.15.119:0/2688693312 wait complete. 2024-01-21T20:00:01.996 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:01 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:00.469587+0000 mgr.smithi119.rqacbm (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T20:00:01.996 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:01 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:01.512151+0000 mon.smithi119 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi119.rqacbm(active, since 2s) 2024-01-21T20:00:02.197 INFO:teuthology.orchestra.run.smithi119.stderr:Fetching dashboard port number... 2024-01-21T20:00:02.197 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp4p8br1fe:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpw2_cu6oe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-21T20:00:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.575+0000 7f1daaa73700 1 Processor -- start 2024-01-21T20:00:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.575+0000 7f1daaa73700 1 -- start start 2024-01-21T20:00:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.575+0000 7f1daaa73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1daaa73700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40fb490 con 0x7f1da40fab80 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da880f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da880f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33902/0 (socket says 172.21.15.119:33902) 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da880f700 1 -- 172.21.15.119:0/268148684 learned_addr learned my addr 172.21.15.119:0/268148684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da880f700 1 -- 172.21.15.119:0/268148684 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40fb5d0 con 0x7f1da40fab80 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da880f700 1 --2- 172.21.15.119:0/268148684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da40fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1d9800b5f0 tx=0x7f1d98006ee0).ready entity=mon.0 client_cookie=b6e3e33afb29a8d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da37fe700 1 -- 172.21.15.119:0/268148684 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d9800d600 con 0x7f1da40fab80 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da37fe700 1 -- 172.21.15.119:0/268148684 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1d9800dbd0 con 0x7f1da40fab80 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.576+0000 7f1da37fe700 1 -- 172.21.15.119:0/268148684 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d98011c00 con 0x7f1da40fab80 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 -- 172.21.15.119:0/268148684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 msgr2=0x7f1da40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 --2- 172.21.15.119:0/268148684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da40fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1d9800b5f0 tx=0x7f1d98006ee0).stop 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 -- 172.21.15.119:0/268148684 shutdown_connections 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 --2- 172.21.15.119:0/268148684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da40fafa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 -- 172.21.15.119:0/268148684 >> 172.21.15.119:0/268148684 conn(0x7f1da40f63c0 msgr2=0x7f1da40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 -- 172.21.15.119:0/268148684 shutdown_connections 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.577+0000 7f1daaa73700 1 -- 172.21.15.119:0/268148684 wait complete. 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1daaa73700 1 Processor -- start 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1daaa73700 1 -- start start 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1daaa73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da4100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1daaa73700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40ff290 con 0x7f1da40fab80 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1da880f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da4100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1da880f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da4100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33908/0 (socket says 172.21.15.119:33908) 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1da880f700 1 -- 172.21.15.119:0/459434661 learned_addr learned my addr 172.21.15.119:0/459434661 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1da880f700 1 -- 172.21.15.119:0/459434661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9800b200 con 0x7f1da40fab80 2024-01-21T20:00:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.578+0000 7f1da880f700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da4100e10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1d9800e040 tx=0x7f1d98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.579+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d9801b410 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.579+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1d9801b9e0 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.579+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da40ff430 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.579+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d9801a820 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.579+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da40ff840 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.579+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1d9800f070 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.580+0000 7f1da17fa700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d8802feb0 0x7f1d88032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.580+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d980415f0 con 0x7f1da40fab80 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.580+0000 7f1da3fff700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d8802feb0 0x7f1d88032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.580+0000 7f1da3fff700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d8802feb0 0x7f1d88032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1d900096a0 tx=0x7f1d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:02.574 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.580+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da4059100 con 0x7f1da40fab80 2024-01-21T20:00:02.576 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.584+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d98016070 con 0x7f1da40fab80 2024-01-21T20:00:02.728 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.736+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f1da4045ad0 con 0x7f1da40fab80 2024-01-21T20:00:02.728 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.736+0000 7f1da17fa700 1 -- 172.21.15.119:0/459434661 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1d980224f0 con 0x7f1da40fab80 2024-01-21T20:00:02.728 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 8443 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d8802feb0 msgr2=0x7f1d88032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d8802feb0 0x7f1d88032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1d900096a0 tx=0x7f1d90006b40).stop 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 msgr2=0x7f1da4100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da4100e10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1d9800e040 tx=0x7f1d98006ee0).stop 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 shutdown_connections 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d8802feb0 0x7f1d88032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 --2- 172.21.15.119:0/459434661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1da40fab80 0x7f1da4100e10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 >> 172.21.15.119:0/459434661 conn(0x7f1da40f63c0 msgr2=0x7f1da4070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 shutdown_connections 2024-01-21T20:00:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:/usr/bin/ceph: 2024-01-21T20:00:02.738+0000 7f1daaa73700 1 -- 172.21.15.119:0/459434661 wait complete. 2024-01-21T20:00:02.999 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:01.622442+0000 mgr.smithi119.rqacbm (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T20:00:02.999 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:01.837609+0000 mon.smithi119 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:03.080 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-21T20:00:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: disabled 2024-01-21T20:00:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: systemctl is-active firewalld.service 2024-01-21T20:00:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:systemctl: inactive 2024-01-21T20:00:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:firewalld.service is not enabled 2024-01-21T20:00:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr:Ceph Dashboard is now available at: 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr: URL: https://smithi119.front.sepia.ceph.com:8443/ 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr: User: admin 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr: Password: vhkm3pzfgq 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.099 INFO:teuthology.orchestra.run.smithi119.stderr:You can access the Ceph CLI with: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: ceph telemetry on 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:For more information see: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:Bootstrap complete. 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:Releasing lock 140176363207144 on /run/cephadm/79b26e34-b897-11ee-95b0-87774f69a715.lock 2024-01-21T20:00:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:Lock 140176363207144 released on /run/cephadm/79b26e34-b897-11ee-95b0-87774f69a715.lock 2024-01-21T20:00:03.132 INFO:tasks.cephadm:Fetching config... 2024-01-21T20:00:03.132 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T20:00:03.132 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-21T20:00:03.150 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-21T20:00:03.150 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T20:00:03.150 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-21T20:00:03.208 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-21T20:00:03.208 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T20:00:03.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/mon.smithi119/keyring of=/dev/stdout 2024-01-21T20:00:03.274 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-21T20:00:03.275 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-01-21T20:00:03.275 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-21T20:00:03.332 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-21T20:00:03.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+YY1WDnw6M9Okf9N8Hn2MO6pKuPeL6ZW86vJySGF/Y7m2sOKYeATKpmCFPC2b30AwTofz5mYoAzvDuLoUwWCV8eVK5bEDzDDFv6yRiBxCBfLTdpFPH/9OrGO7v1RszvcWPGivrKdFlXh+qgShKMTVnFausqGtDpb29LCct2P3V6aw4cUABHena3UfmoRdwV+YvPcX0V+0hA3Hd+vAJdqP4RYfCQ4wxg73nsfZAhzqz2/fKq1hrDrWWHqY4IQBl78Fsm+tvCvOcFhR+olLEj80d99mx8HS6J+/xTSdcPU0kze6P7k3s9F4qRPHQ9dcSfMIabhKsiijswARfn1XS3Ulsl7LLwDjdjC3OC86aUmm0A8n089EB+75/Z+0NjRPMPGUpepLgDTRITBFikag+9sercu2r7joFy59ltDSCeC3kwgkHfTh0aPQGul4HH20op4fdg1q4R+jaFYZNZKh2nMuc/sxr061wwTnj0nvQhffwDZPqQkIP5pVHAV1zyKbNuc= ceph-79b26e34-b897-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T20:00:03.408 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+YY1WDnw6M9Okf9N8Hn2MO6pKuPeL6ZW86vJySGF/Y7m2sOKYeATKpmCFPC2b30AwTofz5mYoAzvDuLoUwWCV8eVK5bEDzDDFv6yRiBxCBfLTdpFPH/9OrGO7v1RszvcWPGivrKdFlXh+qgShKMTVnFausqGtDpb29LCct2P3V6aw4cUABHena3UfmoRdwV+YvPcX0V+0hA3Hd+vAJdqP4RYfCQ4wxg73nsfZAhzqz2/fKq1hrDrWWHqY4IQBl78Fsm+tvCvOcFhR+olLEj80d99mx8HS6J+/xTSdcPU0kze6P7k3s9F4qRPHQ9dcSfMIabhKsiijswARfn1XS3Ulsl7LLwDjdjC3OC86aUmm0A8n089EB+75/Z+0NjRPMPGUpepLgDTRITBFikag+9sercu2r7joFy59ltDSCeC3kwgkHfTh0aPQGul4HH20op4fdg1q4R+jaFYZNZKh2nMuc/sxr061wwTnj0nvQhffwDZPqQkIP5pVHAV1zyKbNuc= ceph-79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T20:00:03.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+YY1WDnw6M9Okf9N8Hn2MO6pKuPeL6ZW86vJySGF/Y7m2sOKYeATKpmCFPC2b30AwTofz5mYoAzvDuLoUwWCV8eVK5bEDzDDFv6yRiBxCBfLTdpFPH/9OrGO7v1RszvcWPGivrKdFlXh+qgShKMTVnFausqGtDpb29LCct2P3V6aw4cUABHena3UfmoRdwV+YvPcX0V+0hA3Hd+vAJdqP4RYfCQ4wxg73nsfZAhzqz2/fKq1hrDrWWHqY4IQBl78Fsm+tvCvOcFhR+olLEj80d99mx8HS6J+/xTSdcPU0kze6P7k3s9F4qRPHQ9dcSfMIabhKsiijswARfn1XS3Ulsl7LLwDjdjC3OC86aUmm0A8n089EB+75/Z+0NjRPMPGUpepLgDTRITBFikag+9sercu2r7joFy59ltDSCeC3kwgkHfTh0aPQGul4HH20op4fdg1q4R+jaFYZNZKh2nMuc/sxr061wwTnj0nvQhffwDZPqQkIP5pVHAV1zyKbNuc= ceph-79b26e34-b897-11ee-95b0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-21T20:00:03.466 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+YY1WDnw6M9Okf9N8Hn2MO6pKuPeL6ZW86vJySGF/Y7m2sOKYeATKpmCFPC2b30AwTofz5mYoAzvDuLoUwWCV8eVK5bEDzDDFv6yRiBxCBfLTdpFPH/9OrGO7v1RszvcWPGivrKdFlXh+qgShKMTVnFausqGtDpb29LCct2P3V6aw4cUABHena3UfmoRdwV+YvPcX0V+0hA3Hd+vAJdqP4RYfCQ4wxg73nsfZAhzqz2/fKq1hrDrWWHqY4IQBl78Fsm+tvCvOcFhR+olLEj80d99mx8HS6J+/xTSdcPU0kze6P7k3s9F4qRPHQ9dcSfMIabhKsiijswARfn1XS3Ulsl7LLwDjdjC3OC86aUmm0A8n089EB+75/Z+0NjRPMPGUpepLgDTRITBFikag+9sercu2r7joFy59ltDSCeC3kwgkHfTh0aPQGul4HH20op4fdg1q4R+jaFYZNZKh2nMuc/sxr061wwTnj0nvQhffwDZPqQkIP5pVHAV1zyKbNuc= ceph-79b26e34-b897-11ee-95b0-87774f69a715 2024-01-21T20:00:03.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-21T20:00:03.933 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:02.737779+0000 mon.smithi119 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.119:0/459434661' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-21T20:00:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 -- 172.21.15.119:0/3288752368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 msgr2=0x7f32a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 --2- 172.21.15.119:0/3288752368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a80f90f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f32a4009fb0 tx=0x7f32a4009b30).stop 2024-01-21T20:00:04.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 -- 172.21.15.119:0/3288752368 shutdown_connections 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 --2- 172.21.15.119:0/3288752368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a80f90f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 -- 172.21.15.119:0/3288752368 >> 172.21.15.119:0/3288752368 conn(0x7f32a80f0b70 msgr2=0x7f32a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 -- 172.21.15.119:0/3288752368 shutdown_connections 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 -- 172.21.15.119:0/3288752368 wait complete. 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 Processor -- start 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.588+0000 7f32af643700 1 -- start start 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32af643700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a8106670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32af643700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a8106b60 con 0x7f32a8060280 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32ad3df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a8106670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32ad3df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a8106670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33926/0 (socket says 172.21.15.119:33926) 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32ad3df700 1 -- 172.21.15.119:0/2857570462 learned_addr learned my addr 172.21.15.119:0/2857570462 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:04.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32ad3df700 1 -- 172.21.15.119:0/2857570462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4005040 con 0x7f32a8060280 2024-01-21T20:00:04.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32ad3df700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a8106670 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f32a4002040 tx=0x7f32a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:04.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a40095f0 con 0x7f32a8060280 2024-01-21T20:00:04.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f32a4007e30 con 0x7f32a8060280 2024-01-21T20:00:04.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8106d60 con 0x7f32a8060280 2024-01-21T20:00:04.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32a4014d20 con 0x7f32a8060280 2024-01-21T20:00:04.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.589+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a81071f0 con 0x7f32a8060280 2024-01-21T20:00:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.590+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f32a4025070 con 0x7f32a8060280 2024-01-21T20:00:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.590+0000 7f329dffb700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f329402ff00 0x7f32940323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.590+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32a40407c0 con 0x7f32a8060280 2024-01-21T20:00:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.590+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a8045ad0 con 0x7f32a8060280 2024-01-21T20:00:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.590+0000 7f32acbde700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f329402ff00 0x7f32940323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:04.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.591+0000 7f32acbde700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f329402ff00 0x7f32940323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f32980096a0 tx=0x7f3298006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:04.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.595+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32a400f300 con 0x7f32a8060280 2024-01-21T20:00:04.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.660+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32a4025300 con 0x7f32a8060280 2024-01-21T20:00:04.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.747+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f32a8107f80 con 0x7f32a8060280 2024-01-21T20:00:04.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.750+0000 7f329dffb700 1 -- 172.21.15.119:0/2857570462 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f32a402d0c0 con 0x7f32a8060280 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f329402ff00 msgr2=0x7f32940323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f329402ff00 0x7f32940323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f32980096a0 tx=0x7f3298006b40).stop 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 msgr2=0x7f32a8106670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a8106670 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f32a4002040 tx=0x7f32a400b040).stop 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 shutdown_connections 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f329402ff00 0x7f32940323c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 --2- 172.21.15.119:0/2857570462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a8060280 0x7f32a8106670 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 >> 172.21.15.119:0/2857570462 conn(0x7f32a80f0b70 msgr2=0x7f32a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 shutdown_connections 2024-01-21T20:00:04.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:04.752+0000 7f32af643700 1 -- 172.21.15.119:0/2857570462 wait complete. 2024-01-21T20:00:05.124 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-01-21T20:00:05.124 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T20:00:05.125 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-01-21T20:00:05.146 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-01-21T20:00:05.146 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-21T20:00:05.207 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-01-21T20:00:05.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph orch host add smithi153 2024-01-21T20:00:06.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:04.662694+0000 mon.smithi119 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi119.rqacbm(active, since 6s) 2024-01-21T20:00:06.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:04.751690+0000 mon.smithi119 (mon.0) 108 : audit [INF] from='client.? 172.21.15.119:0/2857570462' entity='client.admin' 2024-01-21T20:00:08.083 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:06.810344+0000 mon.smithi119 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:08.083 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:07.024326+0000 mon.smithi119 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:08.083 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:07.658835+0000 mon.smithi119 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:11.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:09.880291+0000 mon.smithi119 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:11.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:10.147399+0000 mon.smithi119 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:11.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:10.150201+0000 mon.smithi119 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:11.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:00:10.156445+0000 mgr.smithi119.rqacbm (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi119 on smithi119 2024-01-21T20:00:20.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:18.651108+0000 mgr.smithi119.rqacbm (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:22.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:20.651500+0000 mgr.smithi119.rqacbm (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:24.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:22.651859+0000 mgr.smithi119.rqacbm (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 -- 172.21.15.153:0/258406711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 msgr2=0x7f9b6c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:25.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 --2- 172.21.15.153:0/258406711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f90f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9b5c009fb0 tx=0x7f9b5c009b30).stop 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 -- 172.21.15.153:0/258406711 shutdown_connections 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 --2- 172.21.15.153:0/258406711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f90f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 -- 172.21.15.153:0/258406711 >> 172.21.15.153:0/258406711 conn(0x7f9b6c0f0b70 msgr2=0x7f9b6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 -- 172.21.15.153:0/258406711 shutdown_connections 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.977+0000 7f9b738bb700 1 -- 172.21.15.153:0/258406711 wait complete. 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b738bb700 1 Processor -- start 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b738bb700 1 -- start start 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b738bb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b738bb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6c0f4ff0 con 0x7f9b6c060280 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b71657700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b71657700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41062/0 (socket says 172.21.15.153:41062) 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b71657700 1 -- 172.21.15.153:0/787930036 learned_addr learned my addr 172.21.15.153:0/787930036 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.978+0000 7f9b71657700 1 -- 172.21.15.153:0/787930036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c005040 con 0x7f9b6c060280 2024-01-21T20:00:25.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.979+0000 7f9b71657700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f6b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9b5c013510 tx=0x7f9b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.979+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b5c009320 con 0x7f9b6c060280 2024-01-21T20:00:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.979+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b5c013f00 con 0x7f9b6c060280 2024-01-21T20:00:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.979+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6c0f5190 con 0x7f9b6c060280 2024-01-21T20:00:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.979+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b5c014dc0 con 0x7f9b6c060280 2024-01-21T20:00:25.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.979+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6c0f5620 con 0x7f9b6c060280 2024-01-21T20:00:25.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.980+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b5c014420 con 0x7f9b6c060280 2024-01-21T20:00:25.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.980+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6c059100 con 0x7f9b6c060280 2024-01-21T20:00:25.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.980+0000 7f9b61ffb700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9b5802ff60 0x7f9b58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:25.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.980+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b5c041030 con 0x7f9b6c060280 2024-01-21T20:00:25.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.980+0000 7f9b70e56700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9b5802ff60 0x7f9b58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:25.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.981+0000 7f9b70e56700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9b5802ff60 0x7f9b58032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9b680096a0 tx=0x7f9b68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:25.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:25.985+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b5c0095f0 con 0x7f9b6c060280 2024-01-21T20:00:26.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:24.652243+0000 mgr.smithi119.rqacbm (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:26.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.155+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b6c0f5e70 con 0x7f9b5802ff60 2024-01-21T20:00:26.742 INFO:teuthology.orchestra.run.smithi153.stdout:Added host 'smithi153' 2024-01-21T20:00:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.750+0000 7f9b61ffb700 1 -- 172.21.15.153:0/787930036 <== mgr.14162 v2:172.21.15.119:6800/1043201922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f9b6c0f5e70 con 0x7f9b5802ff60 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9b5802ff60 msgr2=0x7f9b58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9b5802ff60 0x7f9b58032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9b680096a0 tx=0x7f9b68006b40).stop 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 msgr2=0x7f9b6c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f6b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9b5c013510 tx=0x7f9b5c00b040).stop 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 shutdown_connections 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9b5802ff60 0x7f9b58032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:26.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 --2- 172.21.15.153:0/787930036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b6c060280 0x7f9b6c0f6b70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 >> 172.21.15.153:0/787930036 conn(0x7f9b6c0f0b70 msgr2=0x7f9b6c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.752+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 shutdown_connections 2024-01-21T20:00:26.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:26.753+0000 7f9b738bb700 1 -- 172.21.15.153:0/787930036 wait complete. 2024-01-21T20:00:27.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph orch host ls --format=json 2024-01-21T20:00:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:26.157009+0000 mgr.smithi119.rqacbm (mgr.14162) 10 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T20:00:27.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:26.652743+0000 mgr.smithi119.rqacbm (mgr.14162) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:27.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:26.749872+0000 mon.smithi119 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:28.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 -- 172.21.15.153:0/3592127166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 msgr2=0x7f53f80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:28.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 --2- 172.21.15.153:0/3592127166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f80f68a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f53e8009fb0 tx=0x7f53e8009b30).stop 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 -- 172.21.15.153:0/3592127166 shutdown_connections 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 --2- 172.21.15.153:0/3592127166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f80f68a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 -- 172.21.15.153:0/3592127166 >> 172.21.15.153:0/3592127166 conn(0x7f53f80f1c00 msgr2=0x7f53f80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 -- 172.21.15.153:0/3592127166 shutdown_connections 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.259+0000 7f53ffa5f700 1 -- 172.21.15.153:0/3592127166 wait complete. 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53ffa5f700 1 Processor -- start 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53ffa5f700 1 -- start start 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53ffa5f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53ffa5f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8106bc0 con 0x7f53f80f6480 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53fd7fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53fd7fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41078/0 (socket says 172.21.15.153:41078) 2024-01-21T20:00:28.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53fd7fb700 1 -- 172.21.15.153:0/2404646884 learned_addr learned my addr 172.21.15.153:0/2404646884 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53fd7fb700 1 -- 172.21.15.153:0/2404646884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e8005040 con 0x7f53f80f6480 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53fd7fb700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f81066d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f53e8013510 tx=0x7f53e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.260+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e8009320 con 0x7f53f80f6480 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.261+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e8013f00 con 0x7f53f80f6480 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.261+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f8106dc0 con 0x7f53f80f6480 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.261+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e8014d90 con 0x7f53f80f6480 2024-01-21T20:00:28.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.261+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f8107250 con 0x7f53f80f6480 2024-01-21T20:00:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.262+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53e80095f0 con 0x7f53f80f6480 2024-01-21T20:00:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.262+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8045ad0 con 0x7f53f80f6480 2024-01-21T20:00:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.262+0000 7f53edffb700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.262+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53e8040fc0 con 0x7f53f80f6480 2024-01-21T20:00:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.262+0000 7f53fcffa700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.262+0000 7f53fcffa700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f53e402ff60 0x7f53e4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f53f40096a0 tx=0x7f53f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:28.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.266+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53e8014420 con 0x7f53f80f6480 2024-01-21T20:00:28.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.436+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53f80f2e30 con 0x7f53e402ff60 2024-01-21T20:00:28.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.439+0000 7f53edffb700 1 -- 172.21.15.153:0/2404646884 <== mgr.14162 v2:172.21.15.119:6800/1043201922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f53f80f2e30 con 0x7f53e402ff60 2024-01-21T20:00:28.430 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:28.430 INFO:teuthology.orchestra.run.smithi153.stdout:[{"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "smithi153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-01-21T20:00:28.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f53e402ff60 msgr2=0x7f53e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:28.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f53e402ff60 0x7f53e4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f53f40096a0 tx=0x7f53f4006b40).stop 2024-01-21T20:00:28.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 msgr2=0x7f53f81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f81066d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f53e8013510 tx=0x7f53e800b040).stop 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 shutdown_connections 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 --2- 172.21.15.153:0/2404646884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53f80f6480 0x7f53f81066d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 >> 172.21.15.153:0/2404646884 conn(0x7f53f80f1c00 msgr2=0x7f53f80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 shutdown_connections 2024-01-21T20:00:28.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:28.440+0000 7f53ffa5f700 1 -- 172.21.15.153:0/2404646884 wait complete. 2024-01-21T20:00:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:00:26.750989+0000 mgr.smithi119.rqacbm (mgr.14162) 12 : cephadm [INF] Added host smithi153 2024-01-21T20:00:28.810 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-21T20:00:28.811 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph osd crush tunables default 2024-01-21T20:00:29.492 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:28.438906+0000 mgr.smithi119.rqacbm (mgr.14162) 13 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-21T20:00:29.492 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:28.653044+0000 mgr.smithi119.rqacbm (mgr.14162) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:29.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 -- 172.21.15.119:0/4033517171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 msgr2=0x7efcf40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:29.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 --2- 172.21.15.119:0/4033517171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf40f6930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efce8009fb0 tx=0x7efce8009b30).stop 2024-01-21T20:00:29.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 -- 172.21.15.119:0/4033517171 shutdown_connections 2024-01-21T20:00:29.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 --2- 172.21.15.119:0/4033517171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf40f6930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:29.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 -- 172.21.15.119:0/4033517171 >> 172.21.15.119:0/4033517171 conn(0x7efcf40f1c70 msgr2=0x7efcf40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 -- 172.21.15.119:0/4033517171 shutdown_connections 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 -- 172.21.15.119:0/4033517171 wait complete. 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 Processor -- start 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.971+0000 7efcfc58e700 1 -- start start 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfc58e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfc58e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4106c20 con 0x7efcf40f6510 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfa32a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfa32a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56186/0 (socket says 172.21.15.119:56186) 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfa32a700 1 -- 172.21.15.119:0/1085316918 learned_addr learned my addr 172.21.15.119:0/1085316918 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfa32a700 1 -- 172.21.15.119:0/1085316918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce8005040 con 0x7efcf40f6510 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfa32a700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf4106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7efce8013510 tx=0x7efce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce8009320 con 0x7efcf40f6510 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efce8013f00 con 0x7efcf40f6510 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf4106e20 con 0x7efcf40f6510 2024-01-21T20:00:29.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce8014d90 con 0x7efcf40f6510 2024-01-21T20:00:29.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.972+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf41072b0 con 0x7efcf40f6510 2024-01-21T20:00:29.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.973+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efce80263a0 con 0x7efcf40f6510 2024-01-21T20:00:29.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.973+0000 7efce6ffd700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efce002ffb0 0x7efce0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:29.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.973+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efce8041f70 con 0x7efcf40f6510 2024-01-21T20:00:29.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.973+0000 7efcf9b29700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efce002ffb0 0x7efce0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:29.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.973+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4059100 con 0x7efcf40f6510 2024-01-21T20:00:29.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.973+0000 7efcf9b29700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efce002ffb0 0x7efce0032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efcf00096a0 tx=0x7efcf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:29.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:29.977+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efce8012920 con 0x7efcf40f6510 2024-01-21T20:00:30.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.131+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efcf4045ad0 con 0x7efcf40f6510 2024-01-21T20:00:30.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.186+0000 7efce6ffd700 1 -- 172.21.15.119:0/1085316918 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7efce802b030 con 0x7efcf40f6510 2024-01-21T20:00:30.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efce002ffb0 msgr2=0x7efce0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:30.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efce002ffb0 0x7efce0032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efcf00096a0 tx=0x7efcf0006b40).stop 2024-01-21T20:00:30.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 msgr2=0x7efcf4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:30.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf4106730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7efce8013510 tx=0x7efce800b040).stop 2024-01-21T20:00:30.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 shutdown_connections 2024-01-21T20:00:30.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efce002ffb0 0x7efce0032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:30.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 --2- 172.21.15.119:0/1085316918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcf40f6510 0x7efcf4106730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:30.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.187+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 >> 172.21.15.119:0/1085316918 conn(0x7efcf40f1c70 msgr2=0x7efcf40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:30.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.188+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 shutdown_connections 2024-01-21T20:00:30.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-01-21T20:00:30.188+0000 7efcfc58e700 1 -- 172.21.15.119:0/1085316918 wait complete. 2024-01-21T20:00:30.180 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2024-01-21T20:00:30.453 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:30.132761+0000 mon.smithi119 (mon.0) 116 : audit [INF] from='client.? 172.21.15.119:0/1085316918' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-21T20:00:30.544 INFO:tasks.cephadm:Adding mon.smithi119 on smithi119 2024-01-21T20:00:30.544 INFO:tasks.cephadm:Adding mon.smithi153 on smithi153 2024-01-21T20:00:30.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph orch apply mon '2;smithi119:172.21.15.119=smithi119;smithi153:172.21.15.153=smithi153' 2024-01-21T20:00:31.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:30.187454+0000 mon.smithi119 (mon.0) 117 : audit [INF] from='client.? 172.21.15.119:0/1085316918' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-21T20:00:31.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:30.187502+0000 mon.smithi119 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-21T20:00:31.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:30.653309+0000 mgr.smithi119.rqacbm (mgr.14162) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 -- 172.21.15.153:0/3949824432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 msgr2=0x7f7a740f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 --2- 172.21.15.153:0/3949824432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a740f4960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7a5c009fb0 tx=0x7f7a5c009b30).stop 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 -- 172.21.15.153:0/3949824432 shutdown_connections 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 --2- 172.21.15.153:0/3949824432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a740f4960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 -- 172.21.15.153:0/3949824432 >> 172.21.15.153:0/3949824432 conn(0x7f7a740effc0 msgr2=0x7f7a740f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 -- 172.21.15.153:0/3949824432 shutdown_connections 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 -- 172.21.15.153:0/3949824432 wait complete. 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.720+0000 7f7a7359e700 1 Processor -- start 2024-01-21T20:00:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7359e700 1 -- start start 2024-01-21T20:00:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a741064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a74106990 con 0x7f7a740f4540 2024-01-21T20:00:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a741064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a741064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41098/0 (socket says 172.21.15.153:41098) 2024-01-21T20:00:31.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7259c700 1 -- 172.21.15.153:0/2622280006 learned_addr learned my addr 172.21.15.153:0/2622280006 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7259c700 1 -- 172.21.15.153:0/2622280006 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5c005040 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.721+0000 7f7a7259c700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a741064a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7a5c013510 tx=0x7f7a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.722+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c009320 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.722+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a74106b90 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.722+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a5c013f00 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.722+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a5c014d90 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.722+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a74107020 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.723+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a5c0095f0 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.723+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a74045ad0 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.723+0000 7f7a6affd700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7a6002ff60 0x7f7a60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.723+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a5c041000 con 0x7f7a740f4540 2024-01-21T20:00:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.723+0000 7f7a71d9b700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7a6002ff60 0x7f7a60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.723+0000 7f7a71d9b700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7a6002ff60 0x7f7a60032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7a640096a0 tx=0x7f7a64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:31.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.728+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a5c014420 con 0x7f7a740f4540 2024-01-21T20:00:31.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.904+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 --> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi119:172.21.15.119=smithi119;smithi153:172.21.15.153=smithi153", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a740f1210 con 0x7f7a6002ff60 2024-01-21T20:00:31.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.910+0000 7f7a6affd700 1 -- 172.21.15.153:0/2622280006 <== mgr.14162 v2:172.21.15.119:6800/1043201922 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7a740f1210 con 0x7f7a6002ff60 2024-01-21T20:00:31.901 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7a6002ff60 msgr2=0x7f7a60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7a6002ff60 0x7f7a60032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7a640096a0 tx=0x7f7a64006b40).stop 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 msgr2=0x7f7a741064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a741064a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7a5c013510 tx=0x7f7a5c00b040).stop 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 shutdown_connections 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7a6002ff60 0x7f7a60032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 --2- 172.21.15.153:0/2622280006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a740f4540 0x7f7a741064a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 >> 172.21.15.153:0/2622280006 conn(0x7f7a740effc0 msgr2=0x7f7a740f0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 shutdown_connections 2024-01-21T20:00:31.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:31.911+0000 7f7a7359e700 1 -- 172.21.15.153:0/2622280006 wait complete. 2024-01-21T20:00:32.277 DEBUG:teuthology.orchestra.run.smithi153:mon.smithi153> sudo journalctl -f -n 0 -u ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153.service 2024-01-21T20:00:32.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:32.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:32.308 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:-- Logs begin at Sun 2024-01-21 19:49:24 UTC. -- 2024-01-21T20:00:33.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:31.907048+0000 mgr.smithi119.rqacbm (mgr.14162) 16 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi119:172.21.15.119=smithi119;smithi153:172.21.15.153=smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-01-21T20:00:33.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:00:31.908403+0000 mgr.smithi119.rqacbm (mgr.14162) 17 : cephadm [INF] Saving service mon spec with placement smithi119:172.21.15.119=smithi119;smithi153:172.21.15.153=smithi153;count:2 2024-01-21T20:00:33.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:31.911004+0000 mon.smithi119 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:00:33.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:32.653667+0000 mgr.smithi119.rqacbm (mgr.14162) 18 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:33.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 -- 172.21.15.153:0/2880134592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 msgr2=0x7f25900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 --2- 172.21.15.153:0/2880134592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f25900f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f2580009fb0 tx=0x7f2580009b30).stop 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 -- 172.21.15.153:0/2880134592 shutdown_connections 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 --2- 172.21.15.153:0/2880134592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f25900f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 -- 172.21.15.153:0/2880134592 >> 172.21.15.153:0/2880134592 conn(0x7f25900f09a0 msgr2=0x7f25900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 -- 172.21.15.153:0/2880134592 shutdown_connections 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 -- 172.21.15.153:0/2880134592 wait complete. 2024-01-21T20:00:33.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.528+0000 7f258f59e700 1 Processor -- start 2024-01-21T20:00:33.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258f59e700 1 -- start start 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f2590109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590060280 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f2590109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f2590109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54794/0 (socket says 172.21.15.153:54794) 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258e59c700 1 -- 172.21.15.153:0/1711903851 learned_addr learned my addr 172.21.15.153:0/1711903851 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258e59c700 1 -- 172.21.15.153:0/1711903851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2580005040 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.529+0000 7f258e59c700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f2590109170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2580013510 tx=0x7f258000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.530+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2580009320 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.530+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2580013f00 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.530+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2590060480 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.530+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2580014dc0 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.530+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25901098a0 con 0x7f25900f51a0 2024-01-21T20:00:33.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.530+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25800095f0 con 0x7f25900f51a0 2024-01-21T20:00:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.531+0000 7f257effd700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.531+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2590045ad0 con 0x7f25900f51a0 2024-01-21T20:00:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.531+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25800410c0 con 0x7f25900f51a0 2024-01-21T20:00:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.531+0000 7f258dd9b700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.531+0000 7f258dd9b700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f25840096a0 tx=0x7f2584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:33.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.535+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f258001c790 con 0x7f25900f51a0 2024-01-21T20:00:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.739+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25900fabe0 con 0x7f25900f51a0 2024-01-21T20:00:33.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.740+0000 7f257effd700 1 -- 172.21.15.153:0/1711903851 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f258002c090 con 0x7f25900f51a0 2024-01-21T20:00:33.731 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:33.731 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 msgr2=0x7f2578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f25840096a0 tx=0x7f2584006b40).stop 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 msgr2=0x7f2590109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f2590109170 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f2580013510 tx=0x7f258000b040).stop 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 shutdown_connections 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 --2- 172.21.15.153:0/1711903851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f51a0 0x7f2590109170 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 >> 172.21.15.153:0/1711903851 conn(0x7f25900f09a0 msgr2=0x7f25900f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 shutdown_connections 2024-01-21T20:00:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:33.741+0000 7f258f59e700 1 -- 172.21.15.153:0/1711903851 wait complete. 2024-01-21T20:00:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:34.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:33.740962+0000 mon.smithi119 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.153:0/1711903851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:35.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:35.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:35.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:34.653998+0000 mgr.smithi119.rqacbm (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 -- 172.21.15.153:0/4062428637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 msgr2=0x7fb9600f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 --2- 172.21.15.153:0/4062428637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9600f9150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb948009fb0 tx=0x7fb948009b30).stop 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 -- 172.21.15.153:0/4062428637 shutdown_connections 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 --2- 172.21.15.153:0/4062428637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9600f9150 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 -- 172.21.15.153:0/4062428637 >> 172.21.15.153:0/4062428637 conn(0x7fb9600f0b70 msgr2=0x7fb9600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 -- 172.21.15.153:0/4062428637 shutdown_connections 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 -- 172.21.15.153:0/4062428637 wait complete. 2024-01-21T20:00:36.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.275+0000 7fb965d51700 1 Processor -- start 2024-01-21T20:00:36.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb965d51700 1 -- start start 2024-01-21T20:00:36.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb965d51700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9601066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb965d51700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb960106b90 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb95f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9601066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb95f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9601066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54824/0 (socket says 172.21.15.153:54824) 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb95f7fe700 1 -- 172.21.15.153:0/1906747981 learned_addr learned my addr 172.21.15.153:0/1906747981 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb95f7fe700 1 -- 172.21.15.153:0/1906747981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948005040 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.276+0000 7fb95f7fe700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9601066a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb948009830 tx=0x7fb94800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb948009320 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb960106d90 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9480136c0 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb94801ee90 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb960107220 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb948014440 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.277+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb960045ad0 con 0x7fb960060220 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.278+0000 7fb957fff700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb94c02ff10 0x7fb94c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:36.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.278+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb948041300 con 0x7fb960060220 2024-01-21T20:00:36.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.278+0000 7fb95effd700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb94c02ff10 0x7fb94c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:36.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.278+0000 7fb95effd700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb94c02ff10 0x7fb94c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb9500096a0 tx=0x7fb950006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:36.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.282+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb94801c910 con 0x7fb960060220 2024-01-21T20:00:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.488+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9601080d0 con 0x7fb960060220 2024-01-21T20:00:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.488+0000 7fb957fff700 1 -- 172.21.15.153:0/1906747981 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb94801c320 con 0x7fb960060220 2024-01-21T20:00:36.480 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:36.480 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb94c02ff10 msgr2=0x7fb94c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb94c02ff10 0x7fb94c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb9500096a0 tx=0x7fb950006b40).stop 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 msgr2=0x7fb9601066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9601066a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb948009830 tx=0x7fb94800b040).stop 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 shutdown_connections 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb94c02ff10 0x7fb94c0323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 --2- 172.21.15.153:0/1906747981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb960060220 0x7fb9601066a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 >> 172.21.15.153:0/1906747981 conn(0x7fb9600f0b70 msgr2=0x7fb9600f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 shutdown_connections 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:36.490+0000 7fb965d51700 1 -- 172.21.15.153:0/1906747981 wait complete. 2024-01-21T20:00:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:37.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:36.489732+0000 mon.smithi119 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.153:0/1906747981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:37.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:36.654379+0000 mgr.smithi119.rqacbm (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:37.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:37.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.054+0000 7f68c759e700 1 -- 172.21.15.153:0/3905925328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 msgr2=0x7f68c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.054+0000 7f68c759e700 1 --2- 172.21.15.153:0/3905925328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c80f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f68bc009fb0 tx=0x7f68bc009b30).stop 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.054+0000 7f68c759e700 1 -- 172.21.15.153:0/3905925328 shutdown_connections 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.054+0000 7f68c759e700 1 --2- 172.21.15.153:0/3905925328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c80f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.054+0000 7f68c759e700 1 -- 172.21.15.153:0/3905925328 >> 172.21.15.153:0/3905925328 conn(0x7f68c80f09a0 msgr2=0x7f68c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c759e700 1 -- 172.21.15.153:0/3905925328 shutdown_connections 2024-01-21T20:00:39.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c759e700 1 -- 172.21.15.153:0/3905925328 wait complete. 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c759e700 1 Processor -- start 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c759e700 1 -- start start 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c81091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c8060240 con 0x7f68c80f51a0 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c81091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c81091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54846/0 (socket says 172.21.15.153:54846) 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.055+0000 7f68c659c700 1 -- 172.21.15.153:0/3578491318 learned_addr learned my addr 172.21.15.153:0/3578491318 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68c659c700 1 -- 172.21.15.153:0/3578491318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc005040 con 0x7f68c80f51a0 2024-01-21T20:00:39.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68c659c700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c81091e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f68bc013510 tx=0x7f68bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:39.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68bc009320 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c8060440 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68bc013f00 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68bc014dc0 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.056+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c8109890 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.057+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68bc014420 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.057+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c8045ad0 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.057+0000 7f68b6ffd700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f68b002ff60 0x7f68b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.057+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68bc041110 con 0x7f68c80f51a0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.057+0000 7f68c5d9b700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f68b002ff60 0x7f68b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:39.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.058+0000 7f68c5d9b700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f68b002ff60 0x7f68b0032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f68b80096a0 tx=0x7f68b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:39.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.063+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68bc01cb10 con 0x7f68c80f51a0 2024-01-21T20:00:39.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:38.654640+0000 mgr.smithi119.rqacbm (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.266+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68c810a0b0 con 0x7f68c80f51a0 2024-01-21T20:00:39.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.266+0000 7f68b6ffd700 1 -- 172.21.15.153:0/3578491318 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f68bc01cb10 con 0x7f68c80f51a0 2024-01-21T20:00:39.258 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:39.258 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:39.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f68b002ff60 msgr2=0x7f68b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f68b002ff60 0x7f68b0032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f68b80096a0 tx=0x7f68b8006b40).stop 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 msgr2=0x7f68c81091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c81091e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f68bc013510 tx=0x7f68bc00b040).stop 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 shutdown_connections 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f68b002ff60 0x7f68b0032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 --2- 172.21.15.153:0/3578491318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c80f51a0 0x7f68c81091e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 >> 172.21.15.153:0/3578491318 conn(0x7f68c80f09a0 msgr2=0x7f68c80f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 shutdown_connections 2024-01-21T20:00:39.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:39.268+0000 7f68c759e700 1 -- 172.21.15.153:0/3578491318 wait complete. 2024-01-21T20:00:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:40.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:39.267763+0000 mon.smithi119 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.153:0/3578491318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:40.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:40.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:41.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:40.654981+0000 mgr.smithi119.rqacbm (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 -- 172.21.15.153:0/236051262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 msgr2=0x7f1c540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 --2- 172.21.15.153:0/236051262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c540f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f1c48009fb0 tx=0x7f1c48009b30).stop 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 -- 172.21.15.153:0/236051262 shutdown_connections 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 --2- 172.21.15.153:0/236051262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c540f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 -- 172.21.15.153:0/236051262 >> 172.21.15.153:0/236051262 conn(0x7f1c540f09a0 msgr2=0x7f1c540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 -- 172.21.15.153:0/236051262 shutdown_connections 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 -- 172.21.15.153:0/236051262 wait complete. 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 Processor -- start 2024-01-21T20:00:41.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.816+0000 7f1c59835700 1 -- start start 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c59835700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c541091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c59835700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c54060240 con 0x7f1c540f51a0 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c58833700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c541091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c58833700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c541091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54864/0 (socket says 172.21.15.153:54864) 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c58833700 1 -- 172.21.15.153:0/739641206 learned_addr learned my addr 172.21.15.153:0/739641206 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c58833700 1 -- 172.21.15.153:0/739641206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48005040 con 0x7f1c540f51a0 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.817+0000 7f1c58833700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c541091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1c48013510 tx=0x7f1c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:41.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.818+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c48009320 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.818+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c54060440 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.818+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c48013f00 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.818+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c48014dc0 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.818+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c54109860 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.818+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c480095f0 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.819+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c54045ad0 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.819+0000 7f1c517fa700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1c4402ff60 0x7f1c44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.819+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c48041000 con 0x7f1c540f51a0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.819+0000 7f1c53fff700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1c4402ff60 0x7f1c44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:41.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.819+0000 7f1c53fff700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1c4402ff60 0x7f1c44032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1c400096a0 tx=0x7f1c40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:41.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:41.823+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c480268c0 con 0x7f1c540f51a0 2024-01-21T20:00:42.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.027+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c5410a080 con 0x7f1c540f51a0 2024-01-21T20:00:42.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.028+0000 7f1c517fa700 1 -- 172.21.15.153:0/739641206 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c48010230 con 0x7f1c540f51a0 2024-01-21T20:00:42.018 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:42.018 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1c4402ff60 msgr2=0x7f1c44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1c4402ff60 0x7f1c44032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f1c400096a0 tx=0x7f1c40006b40).stop 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 msgr2=0x7f1c541091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c541091b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f1c48013510 tx=0x7f1c4800b040).stop 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 shutdown_connections 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1c4402ff60 0x7f1c44032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:42.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 --2- 172.21.15.153:0/739641206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c540f51a0 0x7f1c541091b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:42.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 >> 172.21.15.153:0/739641206 conn(0x7f1c540f09a0 msgr2=0x7f1c540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:42.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.029+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 shutdown_connections 2024-01-21T20:00:42.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:42.030+0000 7f1c59835700 1 -- 172.21.15.153:0/739641206 wait complete. 2024-01-21T20:00:42.021 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:42.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:42.028320+0000 mon.smithi119 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.153:0/739641206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:43.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:43.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:43.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:42.655352+0000 mgr.smithi119.rqacbm (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 -- 172.21.15.153:0/3225193362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 msgr2=0x7fe83c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 --2- 172.21.15.153:0/3225193362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c0f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe830009fb0 tx=0x7fe830009b30).stop 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 -- 172.21.15.153:0/3225193362 shutdown_connections 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 --2- 172.21.15.153:0/3225193362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c0f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 -- 172.21.15.153:0/3225193362 >> 172.21.15.153:0/3225193362 conn(0x7fe83c0f0b70 msgr2=0x7fe83c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 -- 172.21.15.153:0/3225193362 shutdown_connections 2024-01-21T20:00:44.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.951+0000 7fe841060700 1 -- 172.21.15.153:0/3225193362 wait complete. 2024-01-21T20:00:44.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe841060700 1 Processor -- start 2024-01-21T20:00:44.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe841060700 1 -- start start 2024-01-21T20:00:44.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe841060700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:44.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe841060700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c106c10 con 0x7fe83c0f5410 2024-01-21T20:00:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe83ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe83ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41748/0 (socket says 172.21.15.153:41748) 2024-01-21T20:00:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe83ad9d700 1 -- 172.21.15.153:0/2985968343 learned_addr learned my addr 172.21.15.153:0/2985968343 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.952+0000 7fe83ad9d700 1 -- 172.21.15.153:0/2985968343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe830005040 con 0x7fe83c0f5410 2024-01-21T20:00:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.953+0000 7fe83ad9d700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c106720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe830014000 tx=0x7fe83000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:44.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.953+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8300095f0 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.953+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c106e10 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.953+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe830007e70 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.953+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe830014ea0 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.953+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c1072a0 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.954+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe830025070 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.954+0000 7fe8237fe700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe82402ff60 0x7fe824032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.954+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c045ad0 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.954+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8300409f0 con 0x7fe83c0f5410 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.954+0000 7fe83a59c700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe82402ff60 0x7fe824032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:44.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.954+0000 7fe83a59c700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe82402ff60 0x7fe824032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe82c0096a0 tx=0x7fe82c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:44.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:44.958+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe83000f710 con 0x7fe83c0f5410 2024-01-21T20:00:45.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:44.655724+0000 mgr.smithi119.rqacbm (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:45.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.162+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe83c1080f0 con 0x7fe83c0f5410 2024-01-21T20:00:45.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.162+0000 7fe8237fe700 1 -- 172.21.15.153:0/2985968343 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe830025300 con 0x7fe83c0f5410 2024-01-21T20:00:45.153 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:45.153 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:45.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe82402ff60 msgr2=0x7fe824032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:45.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe82402ff60 0x7fe824032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fe82c0096a0 tx=0x7fe82c006b40).stop 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 msgr2=0x7fe83c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c106720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe830014000 tx=0x7fe83000b040).stop 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 shutdown_connections 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe82402ff60 0x7fe824032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 --2- 172.21.15.153:0/2985968343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe83c0f5410 0x7fe83c106720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 >> 172.21.15.153:0/2985968343 conn(0x7fe83c0f0b70 msgr2=0x7fe83c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 shutdown_connections 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:45.164+0000 7fe841060700 1 -- 172.21.15.153:0/2985968343 wait complete. 2024-01-21T20:00:45.155 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:46.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:45 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:45.162790+0000 mon.smithi119 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.153:0/2985968343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:46.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:46.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:47.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:46.656083+0000 mgr.smithi119.rqacbm (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 -- 172.21.15.153:0/423153886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 msgr2=0x7fdc840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 --2- 172.21.15.153:0/423153886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc840f55c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fdc74009fb0 tx=0x7fdc74009b30).stop 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 -- 172.21.15.153:0/423153886 shutdown_connections 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 --2- 172.21.15.153:0/423153886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc840f55c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 -- 172.21.15.153:0/423153886 >> 172.21.15.153:0/423153886 conn(0x7fdc840f09a0 msgr2=0x7fdc840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 -- 172.21.15.153:0/423153886 shutdown_connections 2024-01-21T20:00:47.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.749+0000 7fdc8359e700 1 -- 172.21.15.153:0/423153886 wait complete. 2024-01-21T20:00:47.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8359e700 1 Processor -- start 2024-01-21T20:00:47.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8359e700 1 -- start start 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc841053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc841058a0 con 0x7fdc840f51a0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc841053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc841053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41776/0 (socket says 172.21.15.153:41776) 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.750+0000 7fdc8259c700 1 -- 172.21.15.153:0/3941659922 learned_addr learned my addr 172.21.15.153:0/3941659922 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc8259c700 1 -- 172.21.15.153:0/3941659922 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc74005040 con 0x7fdc840f51a0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc8259c700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc841053b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fdc74013510 tx=0x7fdc7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc74009320 con 0x7fdc840f51a0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc84105aa0 con 0x7fdc840f51a0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc74013f00 con 0x7fdc840f51a0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc74014dc0 con 0x7fdc840f51a0 2024-01-21T20:00:47.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.751+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc84105f30 con 0x7fdc840f51a0 2024-01-21T20:00:47.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.752+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc740095f0 con 0x7fdc840f51a0 2024-01-21T20:00:47.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.752+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc84045ad0 con 0x7fdc840f51a0 2024-01-21T20:00:47.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.752+0000 7fdc72ffd700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc6c032170 0x7fdc6c034630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:47.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.752+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc740410c0 con 0x7fdc840f51a0 2024-01-21T20:00:47.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.752+0000 7fdc81d9b700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc6c032170 0x7fdc6c034630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:47.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.753+0000 7fdc81d9b700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc6c032170 0x7fdc6c034630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdc780096a0 tx=0x7fdc78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:47.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.757+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc74014420 con 0x7fdc840f51a0 2024-01-21T20:00:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.961+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc84106de0 con 0x7fdc840f51a0 2024-01-21T20:00:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.962+0000 7fdc72ffd700 1 -- 172.21.15.153:0/3941659922 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc74010230 con 0x7fdc840f51a0 2024-01-21T20:00:47.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:47.953 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc6c032170 msgr2=0x7fdc6c034630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc6c032170 0x7fdc6c034630 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fdc780096a0 tx=0x7fdc78006b40).stop 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 msgr2=0x7fdc841053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc841053b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fdc74013510 tx=0x7fdc7400b040).stop 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 shutdown_connections 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc6c032170 0x7fdc6c034630 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 --2- 172.21.15.153:0/3941659922 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc840f51a0 0x7fdc841053b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 >> 172.21.15.153:0/3941659922 conn(0x7fdc840f09a0 msgr2=0x7fdc840f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 shutdown_connections 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:47.964+0000 7fdc8359e700 1 -- 172.21.15.153:0/3941659922 wait complete. 2024-01-21T20:00:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:48.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:47.962452+0000 mon.smithi119 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.153:0/3941659922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:49.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:49.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:49.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:48.656322+0000 mgr.smithi119.rqacbm (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:50.656580+0000 mgr.smithi119.rqacbm (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:53.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:52.656917+0000 mgr.smithi119.rqacbm (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 -- 172.21.15.153:0/2195086245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 msgr2=0x7fdd2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 --2- 172.21.15.153:0/2195086245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c0f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fdd14009fb0 tx=0x7fdd14009b30).stop 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 -- 172.21.15.153:0/2195086245 shutdown_connections 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 --2- 172.21.15.153:0/2195086245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c0f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 -- 172.21.15.153:0/2195086245 >> 172.21.15.153:0/2195086245 conn(0x7fdd2c0f0b70 msgr2=0x7fdd2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 -- 172.21.15.153:0/2195086245 shutdown_connections 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.626+0000 7fdd3156b700 1 -- 172.21.15.153:0/2195086245 wait complete. 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.627+0000 7fdd3156b700 1 Processor -- start 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.627+0000 7fdd3156b700 1 -- start start 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.627+0000 7fdd3156b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:54.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.627+0000 7fdd3156b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c106bd0 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd2affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd2affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:42538/0 (socket says 172.21.15.153:42538) 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd2affd700 1 -- 172.21.15.153:0/3957935678 learned_addr learned my addr 172.21.15.153:0/3957935678 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd2affd700 1 -- 172.21.15.153:0/3957935678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14005040 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd2affd700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c1066e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fdd14013510 tx=0x7fdd1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd14009320 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c106dd0 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c107260 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd14013f00 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.628+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd14014d90 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.629+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd140263a0 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.629+0000 7fdd237fe700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd1802ff60 0x7fdd18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.629+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd14040ff0 con 0x7fdd2c0f5410 2024-01-21T20:00:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.629+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c045ad0 con 0x7fdd2c0f5410 2024-01-21T20:00:54.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.629+0000 7fdd2a7fc700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd1802ff60 0x7fdd18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:00:54.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.630+0000 7fdd2a7fc700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd1802ff60 0x7fdd18032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fdd1c0096a0 tx=0x7fdd1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:00:54.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.634+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd14012920 con 0x7fdd2c0f5410 2024-01-21T20:00:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.840+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd2c0faf50 con 0x7fdd2c0f5410 2024-01-21T20:00:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.841+0000 7fdd237fe700 1 -- 172.21.15.153:0/3957935678 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd1402b020 con 0x7fdd2c0f5410 2024-01-21T20:00:54.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:00:54.833 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:00:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd1802ff60 msgr2=0x7fdd18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd1802ff60 0x7fdd18032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fdd1c0096a0 tx=0x7fdd1c006b40).stop 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 msgr2=0x7fdd2c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c1066e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fdd14013510 tx=0x7fdd1400b040).stop 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 shutdown_connections 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd1802ff60 0x7fdd18032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 --2- 172.21.15.153:0/3957935678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd2c0f5410 0x7fdd2c1066e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 >> 172.21.15.153:0/3957935678 conn(0x7fdd2c0f0b70 msgr2=0x7fdd2c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 shutdown_connections 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:00:54.843+0000 7fdd3156b700 1 -- 172.21.15.153:0/3957935678 wait complete. 2024-01-21T20:00:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:00:55.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:54.657276+0000 mgr.smithi119.rqacbm (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:56.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:55 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:54.842572+0000 mon.smithi119 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.153:0/3957935678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:00:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:56.657656+0000 mgr.smithi119.rqacbm (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:00:58.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:00:58.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:00:59.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:00:58.658030+0000 mgr.smithi119.rqacbm (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:00.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:58.779633+0000 mon.smithi119 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:01:00.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:00:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:00:58.781186+0000 mon.smithi119 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:01:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 -- 172.21.15.153:0/1879569306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 msgr2=0x7ff7640fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 --2- 172.21.15.153:0/1879569306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7640fb550 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff754009fb0 tx=0x7ff754009b30).stop 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 -- 172.21.15.153:0/1879569306 shutdown_connections 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 --2- 172.21.15.153:0/1879569306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7640fb550 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 -- 172.21.15.153:0/1879569306 >> 172.21.15.153:0/1879569306 conn(0x7ff7640f0b70 msgr2=0x7ff7640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 -- 172.21.15.153:0/1879569306 shutdown_connections 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.522+0000 7ff76a8a6700 1 -- 172.21.15.153:0/1879569306 wait complete. 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.523+0000 7ff76a8a6700 1 Processor -- start 2024-01-21T20:01:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.523+0000 7ff76a8a6700 1 -- start start 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.523+0000 7ff76a8a6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7641088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.523+0000 7ff76a8a6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff764108da0 con 0x7ff764060220 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff763fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7641088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff763fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7641088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:42556/0 (socket says 172.21.15.153:42556) 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff763fff700 1 -- 172.21.15.153:0/3728818572 learned_addr learned my addr 172.21.15.153:0/3728818572 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff763fff700 1 -- 172.21.15.153:0/3728818572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff754005040 con 0x7ff764060220 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff763fff700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7641088b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff754013510 tx=0x7ff75400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff754009320 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff754013f00 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff764108fa0 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff754014dc0 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.524+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff764109430 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.525+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff754014420 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.525+0000 7ff760ff9700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff74c02ffb0 0x7ff74c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.525+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff764045ad0 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.525+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7540410b0 con 0x7ff764060220 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.525+0000 7ff7637fe700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff74c02ffb0 0x7ff74c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:00.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.526+0000 7ff7637fe700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff74c02ffb0 0x7ff74c032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff7580096a0 tx=0x7ff758006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:00.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.530+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7540095f0 con 0x7ff764060220 2024-01-21T20:01:00.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.735+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff76410a2e0 con 0x7ff764060220 2024-01-21T20:01:00.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.735+0000 7ff760ff9700 1 -- 172.21.15.153:0/3728818572 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff75401db70 con 0x7ff764060220 2024-01-21T20:01:00.728 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:00.728 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff74c02ffb0 msgr2=0x7ff74c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff74c02ffb0 0x7ff74c032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff7580096a0 tx=0x7ff758006b40).stop 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 msgr2=0x7ff7641088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7641088b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff754013510 tx=0x7ff75400b040).stop 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 shutdown_connections 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff74c02ffb0 0x7ff74c032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 --2- 172.21.15.153:0/3728818572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff764060220 0x7ff7641088b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 >> 172.21.15.153:0/3728818572 conn(0x7ff7640f0b70 msgr2=0x7ff7640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 shutdown_connections 2024-01-21T20:01:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:00.737+0000 7ff76a8a6700 1 -- 172.21.15.153:0/3728818572 wait complete. 2024-01-21T20:01:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:01.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:00.658359+0000 mgr.smithi119.rqacbm (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:01.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:00.736759+0000 mon.smithi119 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.153:0/3728818572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:02.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:02.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:03.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:02.658718+0000 mgr.smithi119.rqacbm (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 -- 172.21.15.153:0/903947831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 msgr2=0x7f48200f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 --2- 172.21.15.153:0/903947831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f48200f8f20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4808009fb0 tx=0x7f4808009b30).stop 2024-01-21T20:01:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 -- 172.21.15.153:0/903947831 shutdown_connections 2024-01-21T20:01:03.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 --2- 172.21.15.153:0/903947831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f48200f8f20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 -- 172.21.15.153:0/903947831 >> 172.21.15.153:0/903947831 conn(0x7f48200f09a0 msgr2=0x7f48200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 -- 172.21.15.153:0/903947831 shutdown_connections 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 -- 172.21.15.153:0/903947831 wait complete. 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.356+0000 7f4825516700 1 Processor -- start 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f4825516700 1 -- start start 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f4825516700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f4820106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f4825516700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48201069f0 con 0x7f48200601e0 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f481ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f4820106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f481ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f4820106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:51986/0 (socket says 172.21.15.153:51986) 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f481ffff700 1 -- 172.21.15.153:0/2817992115 learned_addr learned my addr 172.21.15.153:0/2817992115 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f481ffff700 1 -- 172.21.15.153:0/2817992115 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808005040 con 0x7f48200601e0 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.357+0000 7f481ffff700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f4820106500 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4808013510 tx=0x7f480800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:03.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4808009320 con 0x7f48200601e0 2024-01-21T20:01:03.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4820106bf0 con 0x7f48200601e0 2024-01-21T20:01:03.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4808013f00 con 0x7f48200601e0 2024-01-21T20:01:03.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4808014d90 con 0x7f48200601e0 2024-01-21T20:01:03.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4820107080 con 0x7f48200601e0 2024-01-21T20:01:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48080095f0 con 0x7f48200601e0 2024-01-21T20:01:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.358+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4820045ad0 con 0x7f48200601e0 2024-01-21T20:01:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.359+0000 7f481cff9700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f480c02ff60 0x7f480c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.359+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48080410c0 con 0x7f48200601e0 2024-01-21T20:01:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.359+0000 7f481f7fe700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f480c02ff60 0x7f480c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:03.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.359+0000 7f481f7fe700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f480c02ff60 0x7f480c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f48100096a0 tx=0x7f4810006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:03.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.363+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f480801d7f0 con 0x7f48200601e0 2024-01-21T20:01:03.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.570+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4820107f30 con 0x7f48200601e0 2024-01-21T20:01:03.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.571+0000 7f481cff9700 1 -- 172.21.15.153:0/2817992115 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4808010230 con 0x7f48200601e0 2024-01-21T20:01:03.562 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:03.562 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f480c02ff60 msgr2=0x7f480c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f480c02ff60 0x7f480c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f48100096a0 tx=0x7f4810006b40).stop 2024-01-21T20:01:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 msgr2=0x7f4820106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f4820106500 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4808013510 tx=0x7f480800b040).stop 2024-01-21T20:01:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 shutdown_connections 2024-01-21T20:01:03.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f480c02ff60 0x7f480c032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:03.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 --2- 172.21.15.153:0/2817992115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48200601e0 0x7f4820106500 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:03.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 >> 172.21.15.153:0/2817992115 conn(0x7f48200f09a0 msgr2=0x7f48200f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:03.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 shutdown_connections 2024-01-21T20:01:03.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:03.573+0000 7f4825516700 1 -- 172.21.15.153:0/2817992115 wait complete. 2024-01-21T20:01:03.565 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:04.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:03.571977+0000 mon.smithi119 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.153:0/2817992115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:04.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:04.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:05.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:04.659011+0000 mgr.smithi119.rqacbm (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:06.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.183+0000 7ff7f0ece700 1 -- 172.21.15.153:0/337838351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 msgr2=0x7ff7ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:06.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.183+0000 7ff7f0ece700 1 --2- 172.21.15.153:0/337838351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff7dc009fb0 tx=0x7ff7dc009b30).stop 2024-01-21T20:01:06.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 -- 172.21.15.153:0/337838351 shutdown_connections 2024-01-21T20:01:06.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 --2- 172.21.15.153:0/337838351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:06.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 -- 172.21.15.153:0/337838351 >> 172.21.15.153:0/337838351 conn(0x7ff7ec0f0b70 msgr2=0x7ff7ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:06.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 -- 172.21.15.153:0/337838351 shutdown_connections 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 -- 172.21.15.153:0/337838351 wait complete. 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 Processor -- start 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.184+0000 7ff7f0ece700 1 -- start start 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7f0ece700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7f0ece700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec060240 con 0x7ff7ec0f5410 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7ea59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7ea59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:52002/0 (socket says 172.21.15.153:52002) 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7ea59c700 1 -- 172.21.15.153:0/503208122 learned_addr learned my addr 172.21.15.153:0/503208122 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7ea59c700 1 -- 172.21.15.153:0/503208122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc005040 con 0x7ff7ec0f5410 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7ea59c700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec109430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff7dc013510 tx=0x7ff7dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7dc009320 con 0x7ff7ec0f5410 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ec060440 con 0x7ff7ec0f5410 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7dc013f00 con 0x7ff7ec0f5410 2024-01-21T20:01:06.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7dc014d90 con 0x7ff7ec0f5410 2024-01-21T20:01:06.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.185+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec109ae0 con 0x7ff7ec0f5410 2024-01-21T20:01:06.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.186+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7dc0263a0 con 0x7ff7ec0f5410 2024-01-21T20:01:06.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.186+0000 7ff7daffd700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff7d402ff60 0x7ff7d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:06.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.186+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec045ad0 con 0x7ff7ec0f5410 2024-01-21T20:01:06.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.186+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7dc041020 con 0x7ff7ec0f5410 2024-01-21T20:01:06.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.187+0000 7ff7e9d9b700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff7d402ff60 0x7ff7d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:06.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.187+0000 7ff7e9d9b700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff7d402ff60 0x7ff7d4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff7e00096a0 tx=0x7ff7e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:06.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.191+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7dc012920 con 0x7ff7ec0f5410 2024-01-21T20:01:06.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.396+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7ec0faf50 con 0x7ff7ec0f5410 2024-01-21T20:01:06.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.396+0000 7ff7daffd700 1 -- 172.21.15.153:0/503208122 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff7dc02b020 con 0x7ff7ec0f5410 2024-01-21T20:01:06.388 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:06.388 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff7d402ff60 msgr2=0x7ff7d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff7d402ff60 0x7ff7d4032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff7e00096a0 tx=0x7ff7e0006b40).stop 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 msgr2=0x7ff7ec109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec109430 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff7dc013510 tx=0x7ff7dc00b040).stop 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 shutdown_connections 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff7d402ff60 0x7ff7d4032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 --2- 172.21.15.153:0/503208122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7ec0f5410 0x7ff7ec109430 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 >> 172.21.15.153:0/503208122 conn(0x7ff7ec0f0b70 msgr2=0x7ff7ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 shutdown_connections 2024-01-21T20:01:06.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:06.398+0000 7ff7f0ece700 1 -- 172.21.15.153:0/503208122 wait complete. 2024-01-21T20:01:06.391 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:06.397891+0000 mon.smithi119 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.153:0/503208122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:07.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:06.659294+0000 mgr.smithi119.rqacbm (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:07.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:07.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 -- 172.21.15.153:0/627393734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 msgr2=0x7fd4fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 --2- 172.21.15.153:0/627393734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc0f6930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd4f4009fb0 tx=0x7fd4f4009b30).stop 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 -- 172.21.15.153:0/627393734 shutdown_connections 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 --2- 172.21.15.153:0/627393734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc0f6930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 -- 172.21.15.153:0/627393734 >> 172.21.15.153:0/627393734 conn(0x7fd4fc0f1c70 msgr2=0x7fd4fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 -- 172.21.15.153:0/627393734 shutdown_connections 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.996+0000 7fd501649700 1 -- 172.21.15.153:0/627393734 wait complete. 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd501649700 1 Processor -- start 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd501649700 1 -- start start 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd501649700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd501649700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc106c20 con 0x7fd4fc0f6510 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd4faffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd4faffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:52022/0 (socket says 172.21.15.153:52022) 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.997+0000 7fd4faffd700 1 -- 172.21.15.153:0/1651520500 learned_addr learned my addr 172.21.15.153:0/1651520500 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd4faffd700 1 -- 172.21.15.153:0/1651520500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4005040 con 0x7fd4fc0f6510 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd4faffd700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc106730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd4f4013510 tx=0x7fd4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f4009320 con 0x7fd4fc0f6510 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc106e20 con 0x7fd4fc0f6510 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4f4013f00 con 0x7fd4fc0f6510 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4f4014d90 con 0x7fd4fc0f6510 2024-01-21T20:01:08.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.998+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc1072b0 con 0x7fd4fc0f6510 2024-01-21T20:01:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.999+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4f4014420 con 0x7fd4fc0f6510 2024-01-21T20:01:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.999+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc045ad0 con 0x7fd4fc0f6510 2024-01-21T20:01:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.999+0000 7fd4e37fe700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4e402ff60 0x7fd4e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.999+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4f40410d0 con 0x7fd4fc0f6510 2024-01-21T20:01:08.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:08.999+0000 7fd4fa7fc700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4e402ff60 0x7fd4e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:08.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.000+0000 7fd4fa7fc700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4e402ff60 0x7fd4e4032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd4f00089e0 tx=0x7fd4f0006ec0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.004+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4f40095f0 con 0x7fd4fc0f6510 2024-01-21T20:01:09.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:08.659566+0000 mgr.smithi119.rqacbm (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:09.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.210+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4fc0fbf60 con 0x7fd4fc0f6510 2024-01-21T20:01:09.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.210+0000 7fd4e37fe700 1 -- 172.21.15.153:0/1651520500 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd4f401db70 con 0x7fd4fc0f6510 2024-01-21T20:01:09.202 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:09.202 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4e402ff60 msgr2=0x7fd4e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4e402ff60 0x7fd4e4032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd4f00089e0 tx=0x7fd4f0006ec0).stop 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 msgr2=0x7fd4fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc106730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fd4f4013510 tx=0x7fd4f400b040).stop 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 shutdown_connections 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4e402ff60 0x7fd4e4032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:09.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 --2- 172.21.15.153:0/1651520500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0f6510 0x7fd4fc106730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:09.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 >> 172.21.15.153:0/1651520500 conn(0x7fd4fc0f1c70 msgr2=0x7fd4fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:09.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 shutdown_connections 2024-01-21T20:01:09.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:09.212+0000 7fd501649700 1 -- 172.21.15.153:0/1651520500 wait complete. 2024-01-21T20:01:09.205 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:10.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:09.211981+0000 mon.smithi119 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.153:0/1651520500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:10.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:10.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:11.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:10.659904+0000 mgr.smithi119.rqacbm (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 -- 172.21.15.153:0/1748405323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec0f5410 msgr2=0x7f44ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 --2- 172.21.15.153:0/1748405323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec0f5410 0x7f44ec0f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f44e0009fb0 tx=0x7f44e0009b30).stop 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 -- 172.21.15.153:0/1748405323 shutdown_connections 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 --2- 172.21.15.153:0/1748405323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec0f5410 0x7f44ec0f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 -- 172.21.15.153:0/1748405323 >> 172.21.15.153:0/1748405323 conn(0x7f44ec0f0b70 msgr2=0x7f44ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 -- 172.21.15.153:0/1748405323 shutdown_connections 2024-01-21T20:01:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 -- 172.21.15.153:0/1748405323 wait complete. 2024-01-21T20:01:11.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 Processor -- start 2024-01-21T20:01:11.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.842+0000 7f44f306b700 1 -- start start 2024-01-21T20:01:11.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f306b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 0x7f44ec106c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:11.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f306b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec107130 con 0x7f44ec106820 2024-01-21T20:01:11.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f0e07700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 0x7f44ec106c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:11.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f0e07700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 0x7f44ec106c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:52038/0 (socket says 172.21.15.153:52038) 2024-01-21T20:01:11.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f0e07700 1 -- 172.21.15.153:0/313769220 learned_addr learned my addr 172.21.15.153:0/313769220 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f0e07700 1 -- 172.21.15.153:0/313769220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0005040 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.843+0000 7f44f0e07700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 0x7f44ec106c40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f44e0009f80 tx=0x7f44e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.844+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e00136c0 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.844+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec107330 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.844+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44e00095f0 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.844+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e0014420 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.844+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec107e80 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.844+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44e0025060 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.845+0000 7f44e97fa700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44dc02ff60 0x7f44dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.845+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44e0040e40 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.845+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ec059100 con 0x7f44ec106820 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.845+0000 7f44ebfff700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44dc02ff60 0x7f44dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:11.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.845+0000 7f44ebfff700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44dc02ff60 0x7f44dc032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f44d80096a0 tx=0x7f44d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:11.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:11.849+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44e0028440 con 0x7f44ec106820 2024-01-21T20:01:12.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.053+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44ec045ad0 con 0x7f44ec106820 2024-01-21T20:01:12.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.054+0000 7f44e97fa700 1 -- 172.21.15.153:0/313769220 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f44e001d3a0 con 0x7f44ec106820 2024-01-21T20:01:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:12.045 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44dc02ff60 msgr2=0x7f44dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44dc02ff60 0x7f44dc032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f44d80096a0 tx=0x7f44d8006b40).stop 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 msgr2=0x7f44ec106c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 0x7f44ec106c40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f44e0009f80 tx=0x7f44e0009b30).stop 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 shutdown_connections 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44dc02ff60 0x7f44dc032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 --2- 172.21.15.153:0/313769220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44ec106820 0x7f44ec106c40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 >> 172.21.15.153:0/313769220 conn(0x7f44ec0f0b70 msgr2=0x7f44ec05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 shutdown_connections 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:12.055+0000 7f44f306b700 1 -- 172.21.15.153:0/313769220 wait complete. 2024-01-21T20:01:12.047 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:12.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:12.054971+0000 mon.smithi119 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.153:0/313769220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:13.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:13.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:13.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:12.660238+0000 mgr.smithi119.rqacbm (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- 172.21.15.153:0/3655361035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 msgr2=0x7f98f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 --2- 172.21.15.153:0/3655361035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f00f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f98e0009fb0 tx=0x7f98e0009b30).stop 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- 172.21.15.153:0/3655361035 shutdown_connections 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 --2- 172.21.15.153:0/3655361035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f00f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- 172.21.15.153:0/3655361035 >> 172.21.15.153:0/3655361035 conn(0x7f98f00f0b70 msgr2=0x7f98f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- 172.21.15.153:0/3655361035 shutdown_connections 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- 172.21.15.153:0/3655361035 wait complete. 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 Processor -- start 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- start start 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.621+0000 7f98f8195700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f0107240 con 0x7f98f00f5410 2024-01-21T20:01:14.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f5f31700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f5f31700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:59122/0 (socket says 172.21.15.153:59122) 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f5f31700 1 -- 172.21.15.153:0/2359869155 learned_addr learned my addr 172.21.15.153:0/2359869155 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f5f31700 1 -- 172.21.15.153:0/2359869155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e0005040 con 0x7f98f00f5410 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f5f31700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f010b850 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f98e0013510 tx=0x7f98e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98e0009320 con 0x7f98f00f5410 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f0060220 con 0x7f98f00f5410 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98e0013f00 con 0x7f98f00f5410 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98e0014d90 con 0x7f98f00f5410 2024-01-21T20:01:14.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.622+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f010bd40 con 0x7f98f00f5410 2024-01-21T20:01:14.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.623+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98e00095f0 con 0x7f98f00f5410 2024-01-21T20:01:14.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.623+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f0045ad0 con 0x7f98f00f5410 2024-01-21T20:01:14.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.623+0000 7f98e67fc700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f98dc02ff60 0x7f98dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:14.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.623+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98e0040fe0 con 0x7f98f00f5410 2024-01-21T20:01:14.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.624+0000 7f98f5730700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f98dc02ff60 0x7f98dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:14.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.624+0000 7f98f5730700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f98dc02ff60 0x7f98dc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f98ec0096a0 tx=0x7f98ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:14.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.628+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98e001d7f0 con 0x7f98f00f5410 2024-01-21T20:01:14.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.837+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98f00fae60 con 0x7f98f00f5410 2024-01-21T20:01:14.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.838+0000 7f98e67fc700 1 -- 172.21.15.153:0/2359869155 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f98e0010230 con 0x7f98f00f5410 2024-01-21T20:01:14.829 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:14.829 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f98dc02ff60 msgr2=0x7f98dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f98dc02ff60 0x7f98dc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f98ec0096a0 tx=0x7f98ec006b40).stop 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 msgr2=0x7f98f010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f010b850 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f98e0013510 tx=0x7f98e000b040).stop 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 shutdown_connections 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f98dc02ff60 0x7f98dc032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:14.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 --2- 172.21.15.153:0/2359869155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98f00f5410 0x7f98f010b850 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:14.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.839+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 >> 172.21.15.153:0/2359869155 conn(0x7f98f00f0b70 msgr2=0x7f98f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:14.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.840+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 shutdown_connections 2024-01-21T20:01:14.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:14.840+0000 7f98f8195700 1 -- 172.21.15.153:0/2359869155 wait complete. 2024-01-21T20:01:14.832 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:14.660601+0000 mgr.smithi119.rqacbm (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:16.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:15 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:14.839141+0000 mon.smithi119 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.153:0/2359869155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:16.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:16.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:17.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 -- 172.21.15.153:0/128142679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 msgr2=0x7fd5400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 --2- 172.21.15.153:0/128142679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd5400f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fd530009fb0 tx=0x7fd530009b30).stop 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 -- 172.21.15.153:0/128142679 shutdown_connections 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 --2- 172.21.15.153:0/128142679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd5400f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 -- 172.21.15.153:0/128142679 >> 172.21.15.153:0/128142679 conn(0x7fd5400f0b70 msgr2=0x7fd5400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 -- 172.21.15.153:0/128142679 shutdown_connections 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.419+0000 7fd547ecc700 1 -- 172.21.15.153:0/128142679 wait complete. 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd547ecc700 1 Processor -- start 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd547ecc700 1 -- start start 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd547ecc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd54010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd547ecc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd540107240 con 0x7fd5400f5410 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd545c68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd54010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:17.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd545c68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd54010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:59142/0 (socket says 172.21.15.153:59142) 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.420+0000 7fd545c68700 1 -- 172.21.15.153:0/3739883735 learned_addr learned my addr 172.21.15.153:0/3739883735 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd545c68700 1 -- 172.21.15.153:0/3739883735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530005040 con 0x7fd5400f5410 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd545c68700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd54010b850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd530013510 tx=0x7fd53000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd530009320 con 0x7fd5400f5410 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd530013f00 con 0x7fd5400f5410 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd540060220 con 0x7fd5400f5410 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd530014d90 con 0x7fd5400f5410 2024-01-21T20:01:17.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.421+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54010bd40 con 0x7fd5400f5410 2024-01-21T20:01:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.422+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5300095f0 con 0x7fd5400f5410 2024-01-21T20:01:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.422+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd540045ad0 con 0x7fd5400f5410 2024-01-21T20:01:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.422+0000 7fd5367fc700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd52c02ff60 0x7fd52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.422+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd530041030 con 0x7fd5400f5410 2024-01-21T20:01:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.422+0000 7fd545467700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd52c02ff60 0x7fd52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.423+0000 7fd545467700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd52c02ff60 0x7fd52c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd53c0096a0 tx=0x7fd53c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:17.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.426+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd530014420 con 0x7fd5400f5410 2024-01-21T20:01:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:16.660992+0000 mgr.smithi119.rqacbm (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:17.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.631+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5400fae60 con 0x7fd5400f5410 2024-01-21T20:01:17.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.631+0000 7fd5367fc700 1 -- 172.21.15.153:0/3739883735 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd530010230 con 0x7fd5400f5410 2024-01-21T20:01:17.623 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:17.623 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:17.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd52c02ff60 msgr2=0x7fd52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd52c02ff60 0x7fd52c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd53c0096a0 tx=0x7fd53c006b40).stop 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 msgr2=0x7fd54010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd54010b850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd530013510 tx=0x7fd53000b040).stop 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 shutdown_connections 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd52c02ff60 0x7fd52c032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 --2- 172.21.15.153:0/3739883735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5400f5410 0x7fd54010b850 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 >> 172.21.15.153:0/3739883735 conn(0x7fd5400f0b70 msgr2=0x7fd5400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 shutdown_connections 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:17.633+0000 7fd547ecc700 1 -- 172.21.15.153:0/3739883735 wait complete. 2024-01-21T20:01:17.625 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:18.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:17.632939+0000 mon.smithi119 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.153:0/3739883735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:19.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:19.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:19.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:18.661352+0000 mgr.smithi119.rqacbm (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:20.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 -- 172.21.15.153:0/2006411724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 msgr2=0x7fc7440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:20.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 --2- 172.21.15.153:0/2006411724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc7440f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc738009fb0 tx=0x7fc738009b30).stop 2024-01-21T20:01:20.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 -- 172.21.15.153:0/2006411724 shutdown_connections 2024-01-21T20:01:20.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 --2- 172.21.15.153:0/2006411724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc7440f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:20.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 -- 172.21.15.153:0/2006411724 >> 172.21.15.153:0/2006411724 conn(0x7fc7440f0b70 msgr2=0x7fc7440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:20.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 -- 172.21.15.153:0/2006411724 shutdown_connections 2024-01-21T20:01:20.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.207+0000 7fc74afeb700 1 -- 172.21.15.153:0/2006411724 wait complete. 2024-01-21T20:01:20.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc74afeb700 1 Processor -- start 2024-01-21T20:01:20.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc74afeb700 1 -- start start 2024-01-21T20:01:20.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc74afeb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc744105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:20.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc74afeb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc744105b20 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc748d87700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc744105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc748d87700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc744105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:59162/0 (socket says 172.21.15.153:59162) 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.208+0000 7fc748d87700 1 -- 172.21.15.153:0/3106180769 learned_addr learned my addr 172.21.15.153:0/3106180769 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc748d87700 1 -- 172.21.15.153:0/3106180769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738005040 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc748d87700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc744105630 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc738013510 tx=0x7fc73800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc738009320 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc744105d20 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc738013f00 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc738014d90 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.209+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7441061b0 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.210+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7380263a0 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.210+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc744045ad0 con 0x7fc7440f5410 2024-01-21T20:01:20.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.210+0000 7fc7417fa700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc73402ff60 0x7fc734032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.210+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7380413f0 con 0x7fc7440f5410 2024-01-21T20:01:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.211+0000 7fc743fff700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc73402ff60 0x7fc734032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:20.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.211+0000 7fc743fff700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc73402ff60 0x7fc734032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc7300096a0 tx=0x7fc730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:20.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.215+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7380126f0 con 0x7fc7440f5410 2024-01-21T20:01:20.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.420+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7440fae60 con 0x7fc7440f5410 2024-01-21T20:01:20.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.421+0000 7fc7417fa700 1 -- 172.21.15.153:0/3106180769 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc738012330 con 0x7fc7440f5410 2024-01-21T20:01:20.413 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:20.413 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc73402ff60 msgr2=0x7fc734032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc73402ff60 0x7fc734032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc7300096a0 tx=0x7fc730006b40).stop 2024-01-21T20:01:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 msgr2=0x7fc744105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:20.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc744105630 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc738013510 tx=0x7fc73800b040).stop 2024-01-21T20:01:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 shutdown_connections 2024-01-21T20:01:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc73402ff60 0x7fc734032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 --2- 172.21.15.153:0/3106180769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7440f5410 0x7fc744105630 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 >> 172.21.15.153:0/3106180769 conn(0x7fc7440f0b70 msgr2=0x7fc7440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 shutdown_connections 2024-01-21T20:01:20.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:20.423+0000 7fc74afeb700 1 -- 172.21.15.153:0/3106180769 wait complete. 2024-01-21T20:01:20.417 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:20.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:20.422464+0000 mon.smithi119 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.153:0/3106180769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:21.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:21.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:21.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:20.661674+0000 mgr.smithi119.rqacbm (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 -- 172.21.15.153:0/2920324747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 msgr2=0x7fc6940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 --2- 172.21.15.153:0/2920324747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc6940f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc684009fb0 tx=0x7fc684009b30).stop 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 -- 172.21.15.153:0/2920324747 shutdown_connections 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 --2- 172.21.15.153:0/2920324747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc6940f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 -- 172.21.15.153:0/2920324747 >> 172.21.15.153:0/2920324747 conn(0x7fc6940f0b70 msgr2=0x7fc6940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 -- 172.21.15.153:0/2920324747 shutdown_connections 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.040+0000 7fc69a63a700 1 -- 172.21.15.153:0/2920324747 wait complete. 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.041+0000 7fc69a63a700 1 Processor -- start 2024-01-21T20:01:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.041+0000 7fc69a63a700 1 -- start start 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc69a63a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc694106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc69a63a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc694106c20 con 0x7fc6940f5410 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc693fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc694106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc693fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc694106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:59176/0 (socket says 172.21.15.153:59176) 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc693fff700 1 -- 172.21.15.153:0/1702072750 learned_addr learned my addr 172.21.15.153:0/1702072750 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc693fff700 1 -- 172.21.15.153:0/1702072750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc684005040 con 0x7fc6940f5410 2024-01-21T20:01:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.042+0000 7fc693fff700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc694106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc684013510 tx=0x7fc68400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc684009320 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc684013f00 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc694106e20 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc684014d90 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6941072b0 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc684014420 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.043+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694045ad0 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.044+0000 7fc690ff9700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc68002ff60 0x7fc680032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.044+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6840410c0 con 0x7fc6940f5410 2024-01-21T20:01:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.044+0000 7fc6937fe700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc68002ff60 0x7fc680032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.044+0000 7fc6937fe700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc68002ff60 0x7fc680032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fc67c0096a0 tx=0x7fc67c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.048+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6840095f0 con 0x7fc6940f5410 2024-01-21T20:01:23.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:22.661930+0000 mgr.smithi119.rqacbm (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:23.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.252+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6940fae60 con 0x7fc6940f5410 2024-01-21T20:01:23.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.253+0000 7fc690ff9700 1 -- 172.21.15.153:0/1702072750 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc68401db70 con 0x7fc6940f5410 2024-01-21T20:01:23.244 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:23.245 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc68002ff60 msgr2=0x7fc680032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc68002ff60 0x7fc680032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fc67c0096a0 tx=0x7fc67c006b40).stop 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 msgr2=0x7fc694106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc694106730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc684013510 tx=0x7fc68400b040).stop 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 shutdown_connections 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc68002ff60 0x7fc680032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:23.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 --2- 172.21.15.153:0/1702072750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6940f5410 0x7fc694106730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:23.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.254+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 >> 172.21.15.153:0/1702072750 conn(0x7fc6940f0b70 msgr2=0x7fc6940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:23.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.255+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 shutdown_connections 2024-01-21T20:01:23.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:23.255+0000 7fc69a63a700 1 -- 172.21.15.153:0/1702072750 wait complete. 2024-01-21T20:01:23.247 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:24.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:23.254430+0000 mon.smithi119 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.153:0/1702072750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:24.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:24.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:25.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:24.662256+0000 mgr.smithi119.rqacbm (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:26.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 -- 172.21.15.153:0/627471736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 msgr2=0x7f54440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:26.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 --2- 172.21.15.153:0/627471736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f54440f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f5438009fb0 tx=0x7f5438009b30).stop 2024-01-21T20:01:26.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 -- 172.21.15.153:0/627471736 shutdown_connections 2024-01-21T20:01:26.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 --2- 172.21.15.153:0/627471736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f54440f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:26.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 -- 172.21.15.153:0/627471736 >> 172.21.15.153:0/627471736 conn(0x7f54440f1c70 msgr2=0x7f54440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:26.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 -- 172.21.15.153:0/627471736 shutdown_connections 2024-01-21T20:01:26.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.010+0000 7f544950f700 1 -- 172.21.15.153:0/627471736 wait complete. 2024-01-21T20:01:26.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f544950f700 1 Processor -- start 2024-01-21T20:01:26.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f544950f700 1 -- start start 2024-01-21T20:01:26.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f544950f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f5444106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f544950f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5444106c20 con 0x7f54440f6510 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f5442ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f5444106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f5442ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f5444106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41120/0 (socket says 172.21.15.153:41120) 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.011+0000 7f5442ffd700 1 -- 172.21.15.153:0/3682773187 learned_addr learned my addr 172.21.15.153:0/3682773187 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f5442ffd700 1 -- 172.21.15.153:0/3682773187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438005040 con 0x7f54440f6510 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f5442ffd700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f5444106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5438013510 tx=0x7f543800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438009320 con 0x7f54440f6510 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5444106e20 con 0x7f54440f6510 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5438013f00 con 0x7f54440f6510 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5438014d90 con 0x7f54440f6510 2024-01-21T20:01:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.012+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54441072b0 con 0x7f54440f6510 2024-01-21T20:01:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.013+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54380263a0 con 0x7f54440f6510 2024-01-21T20:01:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.013+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5444045ad0 con 0x7f54440f6510 2024-01-21T20:01:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.013+0000 7f542b7fe700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f542c02ff60 0x7f542c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.013+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54380410a0 con 0x7f54440f6510 2024-01-21T20:01:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.013+0000 7f54427fc700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f542c02ff60 0x7f542c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:26.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.014+0000 7f54427fc700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f542c02ff60 0x7f542c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f54340096a0 tx=0x7f5434006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.018+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54380126f0 con 0x7f54440f6510 2024-01-21T20:01:26.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.222+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54440fbf60 con 0x7f54440f6510 2024-01-21T20:01:26.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.223+0000 7f542b7fe700 1 -- 172.21.15.153:0/3682773187 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5438012330 con 0x7f54440f6510 2024-01-21T20:01:26.215 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:26.215 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:26.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.224+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f542c02ff60 msgr2=0x7f542c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:26.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.224+0000 7f544950f700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f542c02ff60 0x7f542c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f54340096a0 tx=0x7f5434006b40).stop 2024-01-21T20:01:26.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 msgr2=0x7f5444106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f5444106730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5438013510 tx=0x7f543800b040).stop 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 shutdown_connections 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f542c02ff60 0x7f542c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 --2- 172.21.15.153:0/3682773187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54440f6510 0x7f5444106730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 >> 172.21.15.153:0/3682773187 conn(0x7f54440f1c70 msgr2=0x7f54440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 shutdown_connections 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:26.225+0000 7f544950f700 1 -- 172.21.15.153:0/3682773187 wait complete. 2024-01-21T20:01:26.217 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:27.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:26.224386+0000 mon.smithi119 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.153:0/3682773187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:27.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:26.662629+0000 mgr.smithi119.rqacbm (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:27.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:27.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:28.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 -- 172.21.15.153:0/4024541607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 msgr2=0x7fe8940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:28.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 --2- 172.21.15.153:0/4024541607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe8940f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe884009fb0 tx=0x7fe884009b30).stop 2024-01-21T20:01:28.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 -- 172.21.15.153:0/4024541607 shutdown_connections 2024-01-21T20:01:28.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 --2- 172.21.15.153:0/4024541607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe8940f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 -- 172.21.15.153:0/4024541607 >> 172.21.15.153:0/4024541607 conn(0x7fe8940f0b70 msgr2=0x7fe8940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 -- 172.21.15.153:0/4024541607 shutdown_connections 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.901+0000 7fe89c6a4700 1 -- 172.21.15.153:0/4024541607 wait complete. 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89c6a4700 1 Processor -- start 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89c6a4700 1 -- start start 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89c6a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe894105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:28.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89c6a4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe894105b20 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89a440700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe894105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89a440700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe894105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41134/0 (socket says 172.21.15.153:41134) 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89a440700 1 -- 172.21.15.153:0/811181518 learned_addr learned my addr 172.21.15.153:0/811181518 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.902+0000 7fe89a440700 1 -- 172.21.15.153:0/811181518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe884005040 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.903+0000 7fe89a440700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe894105630 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe884013510 tx=0x7fe88400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.903+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe884009320 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.903+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe894105d20 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.903+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe884013f00 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.903+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8941061b0 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.903+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe884014d90 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.904+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe884014420 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.904+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe894045ad0 con 0x7fe8940f5410 2024-01-21T20:01:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.904+0000 7fe88affd700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe8800343c0 0x7fe880036880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:28.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.904+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe884041090 con 0x7fe8940f5410 2024-01-21T20:01:28.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.904+0000 7fe899c3f700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe8800343c0 0x7fe880036880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:28.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.904+0000 7fe899c3f700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe8800343c0 0x7fe880036880 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe8900096a0 tx=0x7fe890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:28.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:28.908+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe88406d2b0 con 0x7fe8940f5410 2024-01-21T20:01:29.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:28.662918+0000 mgr.smithi119.rqacbm (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:29.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.114+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8940fae60 con 0x7fe8940f5410 2024-01-21T20:01:29.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.116+0000 7fe88affd700 1 -- 172.21.15.153:0/811181518 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8840095f0 con 0x7fe8940f5410 2024-01-21T20:01:29.107 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:29.107 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe8800343c0 msgr2=0x7fe880036880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe8800343c0 0x7fe880036880 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe8900096a0 tx=0x7fe890006b40).stop 2024-01-21T20:01:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 msgr2=0x7fe894105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe894105630 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe884013510 tx=0x7fe88400b040).stop 2024-01-21T20:01:29.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 shutdown_connections 2024-01-21T20:01:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe8800343c0 0x7fe880036880 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 --2- 172.21.15.153:0/811181518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8940f5410 0x7fe894105630 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 >> 172.21.15.153:0/811181518 conn(0x7fe8940f0b70 msgr2=0x7fe8940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 shutdown_connections 2024-01-21T20:01:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:29.117+0000 7fe89c6a4700 1 -- 172.21.15.153:0/811181518 wait complete. 2024-01-21T20:01:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:30.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:29.116292+0000 mon.smithi119 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.153:0/811181518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:30.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:30.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:31.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:30.663236+0000 mgr.smithi119.rqacbm (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 -- 172.21.15.153:0/15493179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 msgr2=0x7f8ac00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 --2- 172.21.15.153:0/15493179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac00f90f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8ab4009fb0 tx=0x7f8ab4009b30).stop 2024-01-21T20:01:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 -- 172.21.15.153:0/15493179 shutdown_connections 2024-01-21T20:01:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 --2- 172.21.15.153:0/15493179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac00f90f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 -- 172.21.15.153:0/15493179 >> 172.21.15.153:0/15493179 conn(0x7f8ac00f0b70 msgr2=0x7f8ac00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:31.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 -- 172.21.15.153:0/15493179 shutdown_connections 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.780+0000 7f8ac570e700 1 -- 172.21.15.153:0/15493179 wait complete. 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8ac570e700 1 Processor -- start 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8ac570e700 1 -- start start 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8ac570e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac01066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8ac570e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac0106bb0 con 0x7f8ac0060280 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8abeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac01066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8abeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac01066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41144/0 (socket says 172.21.15.153:41144) 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8abeffd700 1 -- 172.21.15.153:0/2145142206 learned_addr learned my addr 172.21.15.153:0/2145142206 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.781+0000 7f8abeffd700 1 -- 172.21.15.153:0/2145142206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab4005040 con 0x7f8ac0060280 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.782+0000 7f8abeffd700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac01066c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f8ab4013510 tx=0x7f8ab400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.782+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab4009320 con 0x7f8ac0060280 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.782+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac0106db0 con 0x7f8ac0060280 2024-01-21T20:01:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.782+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ab4013f00 con 0x7f8ac0060280 2024-01-21T20:01:31.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.782+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ab4014dc0 con 0x7f8ac0060280 2024-01-21T20:01:31.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.782+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac0107240 con 0x7f8ac0060280 2024-01-21T20:01:31.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.783+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ab40095f0 con 0x7f8ac0060280 2024-01-21T20:01:31.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.783+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac0045ad0 con 0x7f8ac0060280 2024-01-21T20:01:31.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.783+0000 7f8aa77fe700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8aa802ff60 0x7f8aa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:31.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.783+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ab40410c0 con 0x7f8ac0060280 2024-01-21T20:01:31.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.783+0000 7f8abe7fc700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8aa802ff60 0x7f8aa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:31.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.783+0000 7f8abe7fc700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8aa802ff60 0x7f8aa8032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8ab00096a0 tx=0x7f8ab0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:31.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.788+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ab401c790 con 0x7f8ac0060280 2024-01-21T20:01:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.992+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ac01080f0 con 0x7f8ac0060280 2024-01-21T20:01:31.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.992+0000 7f8aa77fe700 1 -- 172.21.15.153:0/2145142206 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ab4010230 con 0x7f8ac0060280 2024-01-21T20:01:31.984 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:31.984 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:31.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8aa802ff60 msgr2=0x7f8aa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:31.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8aa802ff60 0x7f8aa8032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8ab00096a0 tx=0x7f8ab0006b40).stop 2024-01-21T20:01:31.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 msgr2=0x7f8ac01066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:31.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac01066c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f8ab4013510 tx=0x7f8ab400b040).stop 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 shutdown_connections 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8aa802ff60 0x7f8aa8032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 --2- 172.21.15.153:0/2145142206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ac0060280 0x7f8ac01066c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 >> 172.21.15.153:0/2145142206 conn(0x7f8ac00f0b70 msgr2=0x7f8ac00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 shutdown_connections 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:31.994+0000 7f8ac570e700 1 -- 172.21.15.153:0/2145142206 wait complete. 2024-01-21T20:01:31.986 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:32.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:31.993729+0000 mon.smithi119 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.153:0/2145142206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:33.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:33.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:33.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:32.663527+0000 mgr.smithi119.rqacbm (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:34.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 -- 172.21.15.153:0/2953203669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 msgr2=0x7fba800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:34.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 --2- 172.21.15.153:0/2953203669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba800f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fba74009fb0 tx=0x7fba74009b30).stop 2024-01-21T20:01:34.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 -- 172.21.15.153:0/2953203669 shutdown_connections 2024-01-21T20:01:34.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 --2- 172.21.15.153:0/2953203669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba800f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:34.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 -- 172.21.15.153:0/2953203669 >> 172.21.15.153:0/2953203669 conn(0x7fba800f1c70 msgr2=0x7fba800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 -- 172.21.15.153:0/2953203669 shutdown_connections 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.587+0000 7fba84a47700 1 -- 172.21.15.153:0/2953203669 wait complete. 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba84a47700 1 Processor -- start 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba84a47700 1 -- start start 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba84a47700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba80106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba84a47700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80106c20 con 0x7fba800f6510 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba7e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba80106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba7e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba80106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40150/0 (socket says 172.21.15.153:40150) 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.588+0000 7fba7e59c700 1 -- 172.21.15.153:0/3817059170 learned_addr learned my addr 172.21.15.153:0/3817059170 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba7e59c700 1 -- 172.21.15.153:0/3817059170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba74005040 con 0x7fba800f6510 2024-01-21T20:01:34.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba7e59c700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba80106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fba74013510 tx=0x7fba7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:34.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba74009320 con 0x7fba800f6510 2024-01-21T20:01:34.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba80106e20 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba801072b0 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba74013f00 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.589+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba74014d60 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.590+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba740095f0 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.590+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba80045ad0 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.590+0000 7fba6effd700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba6802ff60 0x7fba68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.590+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba740410b0 con 0x7fba800f6510 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.590+0000 7fba7dd9b700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba6802ff60 0x7fba68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:34.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.591+0000 7fba7dd9b700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba6802ff60 0x7fba68032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fba700096a0 tx=0x7fba70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:34.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.595+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba74014420 con 0x7fba800f6510 2024-01-21T20:01:34.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.797+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba800fbf60 con 0x7fba800f6510 2024-01-21T20:01:34.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.798+0000 7fba6effd700 1 -- 172.21.15.153:0/3817059170 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba74012980 con 0x7fba800f6510 2024-01-21T20:01:34.790 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:34.790 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba6802ff60 msgr2=0x7fba68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba6802ff60 0x7fba68032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fba700096a0 tx=0x7fba70006b40).stop 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 msgr2=0x7fba80106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba80106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fba74013510 tx=0x7fba7400b040).stop 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 shutdown_connections 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba6802ff60 0x7fba68032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:34.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 --2- 172.21.15.153:0/3817059170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba800f6510 0x7fba80106730 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:34.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 >> 172.21.15.153:0/3817059170 conn(0x7fba800f1c70 msgr2=0x7fba800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:34.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 shutdown_connections 2024-01-21T20:01:34.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:34.800+0000 7fba84a47700 1 -- 172.21.15.153:0/3817059170 wait complete. 2024-01-21T20:01:34.792 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:35.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:34.663857+0000 mgr.smithi119.rqacbm (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:36.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:34.799645+0000 mon.smithi119 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.153:0/3817059170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:36.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:36.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:37.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 -- 172.21.15.153:0/2276105695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 msgr2=0x7f5cfc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:37.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 --2- 172.21.15.153:0/2276105695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc0f9150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5cf8009fb0 tx=0x7f5cf8009b30).stop 2024-01-21T20:01:37.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 -- 172.21.15.153:0/2276105695 shutdown_connections 2024-01-21T20:01:37.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 --2- 172.21.15.153:0/2276105695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc0f9150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:37.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 -- 172.21.15.153:0/2276105695 >> 172.21.15.153:0/2276105695 conn(0x7f5cfc0f0b70 msgr2=0x7f5cfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:37.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 -- 172.21.15.153:0/2276105695 shutdown_connections 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 -- 172.21.15.153:0/2276105695 wait complete. 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 Processor -- start 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.399+0000 7f5d036c5700 1 -- start start 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d036c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d036c5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc106bd0 con 0x7f5cfc060220 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d01461700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d01461700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40172/0 (socket says 172.21.15.153:40172) 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d01461700 1 -- 172.21.15.153:0/1473915100 learned_addr learned my addr 172.21.15.153:0/1473915100 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d01461700 1 -- 172.21.15.153:0/1473915100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8005040 con 0x7f5cfc060220 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d01461700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5cf8013510 tx=0x7f5cf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf8009320 con 0x7f5cfc060220 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cf8013f00 con 0x7f5cfc060220 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.400+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cfc106dd0 con 0x7f5cfc060220 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.401+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cf8014dc0 con 0x7f5cfc060220 2024-01-21T20:01:37.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.401+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc107260 con 0x7f5cfc060220 2024-01-21T20:01:37.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.401+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cf8014420 con 0x7f5cfc060220 2024-01-21T20:01:37.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.401+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc045ad0 con 0x7f5cfc060220 2024-01-21T20:01:37.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.402+0000 7f5cf1ffb700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ce802ffb0 0x7f5ce8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:37.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.402+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cf8041040 con 0x7f5cfc060220 2024-01-21T20:01:37.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.402+0000 7f5d00c60700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ce802ffb0 0x7f5ce8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:37.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.402+0000 7f5d00c60700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ce802ffb0 0x7f5ce8032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5cec0096a0 tx=0x7f5cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:37.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.408+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cf80095f0 con 0x7f5cfc060220 2024-01-21T20:01:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:36.664180+0000 mgr.smithi119.rqacbm (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:37.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.611+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cfc108110 con 0x7f5cfc060220 2024-01-21T20:01:37.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.612+0000 7f5cf1ffb700 1 -- 172.21.15.153:0/1473915100 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5cf801db70 con 0x7f5cfc060220 2024-01-21T20:01:37.603 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:37.603 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:37.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ce802ffb0 msgr2=0x7f5ce8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:37.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ce802ffb0 0x7f5ce8032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f5cec0096a0 tx=0x7f5cec006b40).stop 2024-01-21T20:01:37.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 msgr2=0x7f5cfc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:37.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5cf8013510 tx=0x7f5cf800b040).stop 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 shutdown_connections 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ce802ffb0 0x7f5ce8032470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 --2- 172.21.15.153:0/1473915100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5cfc060220 0x7f5cfc1066e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 >> 172.21.15.153:0/1473915100 conn(0x7f5cfc0f0b70 msgr2=0x7f5cfc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 shutdown_connections 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:37.614+0000 7f5d036c5700 1 -- 172.21.15.153:0/1473915100 wait complete. 2024-01-21T20:01:37.605 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:38.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:37.612590+0000 mon.smithi119 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.153:0/1473915100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:39.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:39.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:39.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:38.664532+0000 mgr.smithi119.rqacbm (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4164189090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 msgr2=0x7fde840f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 --2- 172.21.15.153:0/4164189090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde840f66c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fde80009fb0 tx=0x7fde80009b30).stop 2024-01-21T20:01:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4164189090 shutdown_connections 2024-01-21T20:01:40.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 --2- 172.21.15.153:0/4164189090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde840f66c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4164189090 >> 172.21.15.153:0/4164189090 conn(0x7fde840f1aa0 msgr2=0x7fde840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4164189090 shutdown_connections 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.255+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4164189090 wait complete. 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.256+0000 7fde8b1d3700 1 Processor -- start 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.256+0000 7fde8b1d3700 1 -- start start 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.256+0000 7fde8b1d3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde841064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:40.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.256+0000 7fde8b1d3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde841069a0 con 0x7fde840f62a0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.256+0000 7fde8a1d1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde841064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde8a1d1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde841064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40190/0 (socket says 172.21.15.153:40190) 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde8a1d1700 1 -- 172.21.15.153:0/4216000785 learned_addr learned my addr 172.21.15.153:0/4216000785 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde8a1d1700 1 -- 172.21.15.153:0/4216000785 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde80005040 con 0x7fde840f62a0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde8a1d1700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde841064b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fde80009830 tx=0x7fde8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde80009320 con 0x7fde840f62a0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde800136c0 con 0x7fde840f62a0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde84106ba0 con 0x7fde840f62a0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8001ee90 con 0x7fde840f62a0 2024-01-21T20:01:40.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.257+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde84107030 con 0x7fde840f62a0 2024-01-21T20:01:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.258+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde8001e420 con 0x7fde840f62a0 2024-01-21T20:01:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.258+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde84045ad0 con 0x7fde840f62a0 2024-01-21T20:01:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.258+0000 7fde7affd700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde7002ff10 0x7fde700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.258+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde80041020 con 0x7fde840f62a0 2024-01-21T20:01:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.258+0000 7fde899d0700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde7002ff10 0x7fde700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:40.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.259+0000 7fde899d0700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde7002ff10 0x7fde700323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fde740096a0 tx=0x7fde74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:40.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.263+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde800095f0 con 0x7fde840f62a0 2024-01-21T20:01:40.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.465+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde84107ee0 con 0x7fde840f62a0 2024-01-21T20:01:40.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.466+0000 7fde7affd700 1 -- 172.21.15.153:0/4216000785 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fde8002c2d0 con 0x7fde840f62a0 2024-01-21T20:01:40.456 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:40.457 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde7002ff10 msgr2=0x7fde700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde7002ff10 0x7fde700323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fde740096a0 tx=0x7fde74006b40).stop 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 msgr2=0x7fde841064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde841064b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fde80009830 tx=0x7fde8000b040).stop 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 shutdown_connections 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde7002ff10 0x7fde700323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:40.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 --2- 172.21.15.153:0/4216000785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde840f62a0 0x7fde841064b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 >> 172.21.15.153:0/4216000785 conn(0x7fde840f1aa0 msgr2=0x7fde840f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 shutdown_connections 2024-01-21T20:01:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:40.468+0000 7fde8b1d3700 1 -- 172.21.15.153:0/4216000785 wait complete. 2024-01-21T20:01:40.459 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:40.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:40.466453+0000 mon.smithi119 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.153:0/4216000785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:41.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:40.664820+0000 mgr.smithi119.rqacbm (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:41.848 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:41.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:43.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:42.665073+0000 mgr.smithi119.rqacbm (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:43.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.117+0000 7ff8e9fba700 1 -- 172.21.15.153:0/495371483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 msgr2=0x7ff8e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.117+0000 7ff8e9fba700 1 --2- 172.21.15.153:0/495371483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e40f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ff8d4009fb0 tx=0x7ff8d4009b30).stop 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.117+0000 7ff8e9fba700 1 -- 172.21.15.153:0/495371483 shutdown_connections 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 --2- 172.21.15.153:0/495371483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e40f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 -- 172.21.15.153:0/495371483 >> 172.21.15.153:0/495371483 conn(0x7ff8e40f0b70 msgr2=0x7ff8e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 -- 172.21.15.153:0/495371483 shutdown_connections 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 -- 172.21.15.153:0/495371483 wait complete. 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 Processor -- start 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 -- start start 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.118+0000 7ff8e9fba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e4105b20 con 0x7ff8e40f5410 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:43.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40196/0 (socket says 172.21.15.153:40196) 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e37fe700 1 -- 172.21.15.153:0/2779006889 learned_addr learned my addr 172.21.15.153:0/2779006889 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e37fe700 1 -- 172.21.15.153:0/2779006889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d4005040 con 0x7ff8e40f5410 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e37fe700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e4105630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff8d4013510 tx=0x7ff8d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d4009320 con 0x7ff8e40f5410 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8d4013f00 con 0x7ff8e40f5410 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e4105d20 con 0x7ff8e40f5410 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8d4014d90 con 0x7ff8e40f5410 2024-01-21T20:01:43.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.119+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e41061b0 con 0x7ff8e40f5410 2024-01-21T20:01:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.120+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8d4014420 con 0x7ff8e40f5410 2024-01-21T20:01:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.120+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e4045ad0 con 0x7ff8e40f5410 2024-01-21T20:01:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.120+0000 7ff8cbfff700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff8cc02ff60 0x7ff8cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.120+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8d40410c0 con 0x7ff8e40f5410 2024-01-21T20:01:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.120+0000 7ff8e2ffd700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff8cc02ff60 0x7ff8cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:43.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.121+0000 7ff8e2ffd700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff8cc02ff60 0x7ff8cc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff8d80096a0 tx=0x7ff8d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:43.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.125+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8d40095f0 con 0x7ff8e40f5410 2024-01-21T20:01:43.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.331+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8e40fae60 con 0x7ff8e40f5410 2024-01-21T20:01:43.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.332+0000 7ff8cbfff700 1 -- 172.21.15.153:0/2779006889 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff8d401db70 con 0x7ff8e40f5410 2024-01-21T20:01:43.322 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:43.322 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff8cc02ff60 msgr2=0x7ff8cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff8cc02ff60 0x7ff8cc032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff8d80096a0 tx=0x7ff8d8006b40).stop 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 msgr2=0x7ff8e4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e4105630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7ff8d4013510 tx=0x7ff8d400b040).stop 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 shutdown_connections 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff8cc02ff60 0x7ff8cc032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 --2- 172.21.15.153:0/2779006889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8e40f5410 0x7ff8e4105630 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 >> 172.21.15.153:0/2779006889 conn(0x7ff8e40f0b70 msgr2=0x7ff8e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 shutdown_connections 2024-01-21T20:01:43.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:43.333+0000 7ff8e9fba700 1 -- 172.21.15.153:0/2779006889 wait complete. 2024-01-21T20:01:43.325 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:44.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:43.332164+0000 mon.smithi119 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.153:0/2779006889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:44.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:44.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:45.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:44.665371+0000 mgr.smithi119.rqacbm (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- 172.21.15.153:0/1266862906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 msgr2=0x7fea6c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 --2- 172.21.15.153:0/1266862906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c0f8f20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fea5c009fb0 tx=0x7fea5c009b30).stop 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- 172.21.15.153:0/1266862906 shutdown_connections 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 --2- 172.21.15.153:0/1266862906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c0f8f20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- 172.21.15.153:0/1266862906 >> 172.21.15.153:0/1266862906 conn(0x7fea6c0f09a0 msgr2=0x7fea6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- 172.21.15.153:0/1266862906 shutdown_connections 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- 172.21.15.153:0/1266862906 wait complete. 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 Processor -- start 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- start start 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.953+0000 7fea72130700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c1069f0 con 0x7fea6c0601e0 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea7112e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea7112e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34818/0 (socket says 172.21.15.153:34818) 2024-01-21T20:01:45.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea7112e700 1 -- 172.21.15.153:0/1618892257 learned_addr learned my addr 172.21.15.153:0/1618892257 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea7112e700 1 -- 172.21.15.153:0/1618892257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c005040 con 0x7fea6c0601e0 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea7112e700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c106500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fea5c013510 tx=0x7fea5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea5c009320 con 0x7fea6c0601e0 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c106bf0 con 0x7fea6c0601e0 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea5c013f00 con 0x7fea6c0601e0 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea5c014d90 con 0x7fea6c0601e0 2024-01-21T20:01:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.954+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c107080 con 0x7fea6c0601e0 2024-01-21T20:01:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.955+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea5c014420 con 0x7fea6c0601e0 2024-01-21T20:01:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.955+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c045ad0 con 0x7fea6c0601e0 2024-01-21T20:01:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.955+0000 7fea61ffb700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea5802ff60 0x7fea58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.955+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea5c041110 con 0x7fea6c0601e0 2024-01-21T20:01:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.955+0000 7fea7092d700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea5802ff60 0x7fea58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.956+0000 7fea7092d700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea5802ff60 0x7fea58032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fea680096a0 tx=0x7fea68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:45.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:45.960+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea5c0095f0 con 0x7fea6c0601e0 2024-01-21T20:01:46.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.165+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea6c107ed0 con 0x7fea6c0601e0 2024-01-21T20:01:46.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.165+0000 7fea61ffb700 1 -- 172.21.15.153:0/1618892257 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea5c01db70 con 0x7fea6c0601e0 2024-01-21T20:01:46.156 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:46.156 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:46.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea5802ff60 msgr2=0x7fea58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:46.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea5802ff60 0x7fea58032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fea680096a0 tx=0x7fea68006b40).stop 2024-01-21T20:01:46.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 msgr2=0x7fea6c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:46.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c106500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fea5c013510 tx=0x7fea5c00b040).stop 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 shutdown_connections 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea5802ff60 0x7fea58032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 --2- 172.21.15.153:0/1618892257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea6c0601e0 0x7fea6c106500 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 >> 172.21.15.153:0/1618892257 conn(0x7fea6c0f09a0 msgr2=0x7fea6c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 shutdown_connections 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:46.167+0000 7fea72130700 1 -- 172.21.15.153:0/1618892257 wait complete. 2024-01-21T20:01:46.158 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:47.563 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:47.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:46.165733+0000 mon.smithi119 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.153:0/1618892257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:46.665654+0000 mgr.smithi119.rqacbm (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 -- 172.21.15.153:0/3022801625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 msgr2=0x7f5df00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 --2- 172.21.15.153:0/3022801625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df00f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5de0009fb0 tx=0x7f5de0009b30).stop 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 -- 172.21.15.153:0/3022801625 shutdown_connections 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 --2- 172.21.15.153:0/3022801625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df00f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 -- 172.21.15.153:0/3022801625 >> 172.21.15.153:0/3022801625 conn(0x7f5df00f0b70 msgr2=0x7f5df00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 -- 172.21.15.153:0/3022801625 shutdown_connections 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.770+0000 7f5df63ca700 1 -- 172.21.15.153:0/3022801625 wait complete. 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5df63ca700 1 Processor -- start 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5df63ca700 1 -- start start 2024-01-21T20:01:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5df63ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5df63ca700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df0106c10 con 0x7f5df00f5410 2024-01-21T20:01:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5deffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5deffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34826/0 (socket says 172.21.15.153:34826) 2024-01-21T20:01:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.771+0000 7f5deffff700 1 -- 172.21.15.153:0/1647922511 learned_addr learned my addr 172.21.15.153:0/1647922511 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5deffff700 1 -- 172.21.15.153:0/1647922511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0005040 con 0x7f5df00f5410 2024-01-21T20:01:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5deffff700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df0106720 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5de0013510 tx=0x7f5de000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5de0009320 con 0x7f5df00f5410 2024-01-21T20:01:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df0106e10 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5de0013f00 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5de0014d90 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.772+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df01072a0 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.773+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5de00263a0 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.773+0000 7f5ded7fa700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ddc02ffb0 0x7f5ddc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.773+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5de0041f80 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.773+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df0059100 con 0x7f5df00f5410 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.773+0000 7f5de7fff700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ddc02ffb0 0x7f5ddc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.774+0000 7f5de7fff700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ddc02ffb0 0x7f5ddc032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5dd80096a0 tx=0x7f5dd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:48.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.778+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5de0012920 con 0x7f5df00f5410 2024-01-21T20:01:48.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.982+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5df00fae60 con 0x7f5df00f5410 2024-01-21T20:01:48.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.983+0000 7f5ded7fa700 1 -- 172.21.15.153:0/1647922511 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5de002b020 con 0x7f5df00f5410 2024-01-21T20:01:48.973 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:48.974 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.984+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ddc02ffb0 msgr2=0x7f5ddc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ddc02ffb0 0x7f5ddc032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5dd80096a0 tx=0x7f5dd8006b40).stop 2024-01-21T20:01:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 msgr2=0x7f5df0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df0106720 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5de0013510 tx=0x7f5de000b040).stop 2024-01-21T20:01:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 shutdown_connections 2024-01-21T20:01:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f5ddc02ffb0 0x7f5ddc032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:48.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 --2- 172.21.15.153:0/1647922511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5df00f5410 0x7f5df0106720 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:48.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 >> 172.21.15.153:0/1647922511 conn(0x7f5df00f0b70 msgr2=0x7f5df005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:48.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 shutdown_connections 2024-01-21T20:01:48.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:48.985+0000 7f5df63ca700 1 -- 172.21.15.153:0/1647922511 wait complete. 2024-01-21T20:01:48.976 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:49.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:48.665996+0000 mgr.smithi119.rqacbm (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:50.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:48.983435+0000 mon.smithi119 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.153:0/1647922511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:50.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:50.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:50.666345+0000 mgr.smithi119.rqacbm (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:51.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.599+0000 7f69e0686700 1 -- 172.21.15.153:0/1303462517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 msgr2=0x7f69d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.599+0000 7f69e0686700 1 --2- 172.21.15.153:0/1303462517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d80f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f69cc009fb0 tx=0x7f69cc009b30).stop 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.599+0000 7f69e0686700 1 -- 172.21.15.153:0/1303462517 shutdown_connections 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.599+0000 7f69e0686700 1 --2- 172.21.15.153:0/1303462517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d80f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.599+0000 7f69e0686700 1 -- 172.21.15.153:0/1303462517 >> 172.21.15.153:0/1303462517 conn(0x7f69d80f0b70 msgr2=0x7f69d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.599+0000 7f69e0686700 1 -- 172.21.15.153:0/1303462517 shutdown_connections 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69e0686700 1 -- 172.21.15.153:0/1303462517 wait complete. 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69e0686700 1 Processor -- start 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69e0686700 1 -- start start 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69e0686700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69e0686700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d8106bd0 con 0x7f69d80f5410 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69de422700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.600+0000 7f69de422700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34838/0 (socket says 172.21.15.153:34838) 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69de422700 1 -- 172.21.15.153:0/1798940014 learned_addr learned my addr 172.21.15.153:0/1798940014 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:51.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69de422700 1 -- 172.21.15.153:0/1798940014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc005040 con 0x7f69d80f5410 2024-01-21T20:01:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69de422700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d81066e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f69cc014000 tx=0x7f69cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc0095f0 con 0x7f69d80f5410 2024-01-21T20:01:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69cc007e70 con 0x7f69d80f5410 2024-01-21T20:01:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d8106dd0 con 0x7f69d80f5410 2024-01-21T20:01:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69cc014ea0 con 0x7f69d80f5410 2024-01-21T20:01:51.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.601+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d8107260 con 0x7f69d80f5410 2024-01-21T20:01:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.602+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69cc025070 con 0x7f69d80f5410 2024-01-21T20:01:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.602+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d8045ad0 con 0x7f69d80f5410 2024-01-21T20:01:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.602+0000 7f69caffd700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69c402ff60 0x7f69c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.602+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69cc0409f0 con 0x7f69d80f5410 2024-01-21T20:01:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.602+0000 7f69ddc21700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69c402ff60 0x7f69c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:51.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.603+0000 7f69ddc21700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69c402ff60 0x7f69c4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f69d40096a0 tx=0x7f69d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:51.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.607+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69cc00f4e0 con 0x7f69d80f5410 2024-01-21T20:01:51.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.811+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69d80fae60 con 0x7f69d80f5410 2024-01-21T20:01:51.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.812+0000 7f69caffd700 1 -- 172.21.15.153:0/1798940014 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f69cc00f300 con 0x7f69d80f5410 2024-01-21T20:01:51.803 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:51.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69c402ff60 msgr2=0x7f69c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69c402ff60 0x7f69c4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f69d40096a0 tx=0x7f69d4006b40).stop 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 msgr2=0x7f69d81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d81066e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f69cc014000 tx=0x7f69cc00b040).stop 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 shutdown_connections 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f69c402ff60 0x7f69c4032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 --2- 172.21.15.153:0/1798940014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69d80f5410 0x7f69d81066e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 >> 172.21.15.153:0/1798940014 conn(0x7f69d80f0b70 msgr2=0x7f69d805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 shutdown_connections 2024-01-21T20:01:51.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:51.813+0000 7f69e0686700 1 -- 172.21.15.153:0/1798940014 wait complete. 2024-01-21T20:01:51.806 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:52.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:51.813168+0000 mon.smithi119 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.153:0/1798940014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:53.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:53.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:53.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:52.666709+0000 mgr.smithi119.rqacbm (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/2391239663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 msgr2=0x7fe2e40f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 --2- 172.21.15.153:0/2391239663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e40f6ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe2cc009fb0 tx=0x7fe2cc009b30).stop 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/2391239663 shutdown_connections 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 --2- 172.21.15.153:0/2391239663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e40f6ea0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/2391239663 >> 172.21.15.153:0/2391239663 conn(0x7fe2e40f02e0 msgr2=0x7fe2e40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/2391239663 shutdown_connections 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.391+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/2391239663 wait complete. 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e8fc7700 1 Processor -- start 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e8fc7700 1 -- start start 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e8fc7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e410b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e8fc7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e4107220 con 0x7fe2e40f6a80 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e410b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e410b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56688/0 (socket says 172.21.15.153:56688) 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.392+0000 7fe2e259c700 1 -- 172.21.15.153:0/3211448049 learned_addr learned my addr 172.21.15.153:0/3211448049 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2e259c700 1 -- 172.21.15.153:0/3211448049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc005040 con 0x7fe2e40f6a80 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2e259c700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e410b830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fe2cc013510 tx=0x7fe2cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2cc009320 con 0x7fe2e40f6a80 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2cc013f00 con 0x7fe2e40f6a80 2024-01-21T20:01:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e40f5310 con 0x7fe2e40f6a80 2024-01-21T20:01:54.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2cc014dc0 con 0x7fe2e40f6a80 2024-01-21T20:01:54.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.393+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e410bd20 con 0x7fe2e40f6a80 2024-01-21T20:01:54.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.394+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2cc014420 con 0x7fe2e40f6a80 2024-01-21T20:01:54.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.394+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e4045ad0 con 0x7fe2e40f6a80 2024-01-21T20:01:54.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.394+0000 7fe2daffd700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2d002ff60 0x7fe2d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:54.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.394+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2cc0410f0 con 0x7fe2e40f6a80 2024-01-21T20:01:54.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.394+0000 7fe2e1d9b700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2d002ff60 0x7fe2d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:54.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.395+0000 7fe2e1d9b700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2d002ff60 0x7fe2d0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe2d40096a0 tx=0x7fe2d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:54.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.399+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2cc0095f0 con 0x7fe2e40f6a80 2024-01-21T20:01:54.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.604+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2e410c4e0 con 0x7fe2e40f6a80 2024-01-21T20:01:54.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.604+0000 7fe2daffd700 1 -- 172.21.15.153:0/3211448049 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe2cc01cb10 con 0x7fe2e40f6a80 2024-01-21T20:01:54.596 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:54.596 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2d002ff60 msgr2=0x7fe2d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2d002ff60 0x7fe2d0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe2d40096a0 tx=0x7fe2d4006b40).stop 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 msgr2=0x7fe2e410b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e410b830 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fe2cc013510 tx=0x7fe2cc00b040).stop 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 shutdown_connections 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2d002ff60 0x7fe2d0032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 --2- 172.21.15.153:0/3211448049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2e40f6a80 0x7fe2e410b830 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 >> 172.21.15.153:0/3211448049 conn(0x7fe2e40f02e0 msgr2=0x7fe2e40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 shutdown_connections 2024-01-21T20:01:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:54.606+0000 7fe2e8fc7700 1 -- 172.21.15.153:0/3211448049 wait complete. 2024-01-21T20:01:54.599 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:55.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:54.605996+0000 mon.smithi119 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.153:0/3211448049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:55.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:55.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:56.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:55 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:54.667045+0000 mgr.smithi119.rqacbm (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:57.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 -- 172.21.15.153:0/381389000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 msgr2=0x7ff2880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 --2- 172.21.15.153:0/381389000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff278009fb0 tx=0x7ff278009b30).stop 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 -- 172.21.15.153:0/381389000 shutdown_connections 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 --2- 172.21.15.153:0/381389000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 -- 172.21.15.153:0/381389000 >> 172.21.15.153:0/381389000 conn(0x7ff2880f0b70 msgr2=0x7ff2880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 -- 172.21.15.153:0/381389000 shutdown_connections 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.325+0000 7ff28f662700 1 -- 172.21.15.153:0/381389000 wait complete. 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28f662700 1 Processor -- start 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28f662700 1 -- start start 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28f662700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28f662700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2880fb820 con 0x7ff2880f5410 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28d3fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:57.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28d3fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56692/0 (socket says 172.21.15.153:56692) 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.326+0000 7ff28d3fe700 1 -- 172.21.15.153:0/867094500 learned_addr learned my addr 172.21.15.153:0/867094500 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff28d3fe700 1 -- 172.21.15.153:0/867094500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278005040 con 0x7ff2880f5410 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff28d3fe700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880fd3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff278013510 tx=0x7ff27800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff278009320 con 0x7ff2880f5410 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff278013f00 con 0x7ff2880f5410 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2880fb9c0 con 0x7ff2880f5410 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff278014d90 con 0x7ff2880f5410 2024-01-21T20:01:57.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.327+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2880fbe50 con 0x7ff2880f5410 2024-01-21T20:01:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.328+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff278014420 con 0x7ff2880f5410 2024-01-21T20:01:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.328+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff288059100 con 0x7ff2880f5410 2024-01-21T20:01:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.328+0000 7ff27dffb700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff27402ff60 0x7ff274032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:01:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.328+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff278041110 con 0x7ff2880f5410 2024-01-21T20:01:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.328+0000 7ff28cbfd700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff27402ff60 0x7ff274032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:01:57.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.329+0000 7ff28cbfd700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff27402ff60 0x7ff274032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff2840096a0 tx=0x7ff284006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:01:57.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.333+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2780095f0 con 0x7ff2880f5410 2024-01-21T20:01:57.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.538+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff288045ad0 con 0x7ff2880f5410 2024-01-21T20:01:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.538+0000 7ff27dffb700 1 -- 172.21.15.153:0/867094500 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff27801db70 con 0x7ff2880f5410 2024-01-21T20:01:57.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:01:57.530 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff27402ff60 msgr2=0x7ff274032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff27402ff60 0x7ff274032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff2840096a0 tx=0x7ff284006b40).stop 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 msgr2=0x7ff2880fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880fd3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff278013510 tx=0x7ff27800b040).stop 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 shutdown_connections 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff27402ff60 0x7ff274032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 --2- 172.21.15.153:0/867094500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2880f5410 0x7ff2880fd3a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 >> 172.21.15.153:0/867094500 conn(0x7ff2880f0b70 msgr2=0x7ff2880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 shutdown_connections 2024-01-21T20:01:57.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:01:57.540+0000 7ff28f662700 1 -- 172.21.15.153:0/867094500 wait complete. 2024-01-21T20:01:57.533 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:01:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:56.667394+0000 mgr.smithi119.rqacbm (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:58.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:57.539952+0000 mon.smithi119 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.153:0/867094500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:01:58.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:01:58.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:01:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:01:58.667665+0000 mgr.smithi119.rqacbm (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:01:59.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:58.780571+0000 mon.smithi119 (mon.0) 150 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:01:59.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:01:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:01:58.782029+0000 mon.smithi119 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:02:00.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 -- 172.21.15.153:0/3082206732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 msgr2=0x7f0ff00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:00.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 --2- 172.21.15.153:0/3082206732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff00f55c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0fe4009fb0 tx=0x7f0fe4009b30).stop 2024-01-21T20:02:00.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 -- 172.21.15.153:0/3082206732 shutdown_connections 2024-01-21T20:02:00.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 --2- 172.21.15.153:0/3082206732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff00f55c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 -- 172.21.15.153:0/3082206732 >> 172.21.15.153:0/3082206732 conn(0x7f0ff00f09a0 msgr2=0x7f0ff00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 -- 172.21.15.153:0/3082206732 shutdown_connections 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.161+0000 7f0fef59e700 1 -- 172.21.15.153:0/3082206732 wait complete. 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fef59e700 1 Processor -- start 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fef59e700 1 -- start start 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fef59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fef59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff0060220 con 0x7f0ff00f51a0 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fee59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fee59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56704/0 (socket says 172.21.15.153:56704) 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.162+0000 7f0fee59c700 1 -- 172.21.15.153:0/1583332324 learned_addr learned my addr 172.21.15.153:0/1583332324 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fee59c700 1 -- 172.21.15.153:0/1583332324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe4005040 con 0x7f0ff00f51a0 2024-01-21T20:02:00.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fee59c700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff01091d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0fe4009830 tx=0x7f0fe400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fe4009320 con 0x7f0ff00f51a0 2024-01-21T20:02:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff0060420 con 0x7f0ff00f51a0 2024-01-21T20:02:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fe40136c0 con 0x7f0ff00f51a0 2024-01-21T20:02:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fe401ee90 con 0x7f0ff00f51a0 2024-01-21T20:02:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.163+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff0109880 con 0x7f0ff00f51a0 2024-01-21T20:02:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.164+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fe4014440 con 0x7f0ff00f51a0 2024-01-21T20:02:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.164+0000 7f0fdeffd700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0fd802ff10 0x7f0fd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.164+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fe4040fd0 con 0x7f0ff00f51a0 2024-01-21T20:02:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.164+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff0059100 con 0x7f0ff00f51a0 2024-01-21T20:02:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.164+0000 7f0fedd9b700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0fd802ff10 0x7f0fd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.165+0000 7f0fedd9b700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0fd802ff10 0x7f0fd80323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0fe00096a0 tx=0x7f0fe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:00.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.169+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fe40273f0 con 0x7f0ff00f51a0 2024-01-21T20:02:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.375+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ff00fabe0 con 0x7f0ff00f51a0 2024-01-21T20:02:00.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.376+0000 7f0fdeffd700 1 -- 172.21.15.153:0/1583332324 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0fe401c320 con 0x7f0ff00f51a0 2024-01-21T20:02:00.368 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:00.368 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:00.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0fd802ff10 msgr2=0x7f0fd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:00.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0fd802ff10 0x7f0fd80323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0fe00096a0 tx=0x7f0fe0006b40).stop 2024-01-21T20:02:00.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 msgr2=0x7f0ff01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:00.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff01091d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0fe4009830 tx=0x7f0fe400b040).stop 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 shutdown_connections 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0fd802ff10 0x7f0fd80323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 --2- 172.21.15.153:0/1583332324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ff00f51a0 0x7f0ff01091d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 >> 172.21.15.153:0/1583332324 conn(0x7f0ff00f09a0 msgr2=0x7f0ff00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 shutdown_connections 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:00.378+0000 7f0fef59e700 1 -- 172.21.15.153:0/1583332324 wait complete. 2024-01-21T20:02:00.370 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:00.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:00.377587+0000 mon.smithi119 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.153:0/1583332324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:01.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:01.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:01.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:01 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:00.667998+0000 mgr.smithi119.rqacbm (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 -- 172.21.15.153:0/3353656356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 msgr2=0x7fce5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 --2- 172.21.15.153:0/3353656356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c0f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fce50009fb0 tx=0x7fce50009b30).stop 2024-01-21T20:02:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 -- 172.21.15.153:0/3353656356 shutdown_connections 2024-01-21T20:02:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 --2- 172.21.15.153:0/3353656356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c0f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 -- 172.21.15.153:0/3353656356 >> 172.21.15.153:0/3353656356 conn(0x7fce5c0f0b70 msgr2=0x7fce5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 -- 172.21.15.153:0/3353656356 shutdown_connections 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 -- 172.21.15.153:0/3353656356 wait complete. 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 Processor -- start 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.983+0000 7fce62b81700 1 -- start start 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce62b81700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce62b81700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5c106c10 con 0x7fce5c0f5410 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce6091d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce6091d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56714/0 (socket says 172.21.15.153:56714) 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce6091d700 1 -- 172.21.15.153:0/3978135791 learned_addr learned my addr 172.21.15.153:0/3978135791 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce6091d700 1 -- 172.21.15.153:0/3978135791 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50005040 con 0x7fce5c0f5410 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce6091d700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c106720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fce50013510 tx=0x7fce5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce50009320 con 0x7fce5c0f5410 2024-01-21T20:02:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce50013f00 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5c106e10 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.984+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce50014d90 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.985+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5c1072a0 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.985+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce500263a0 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.985+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5c045ad0 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.985+0000 7fce597fa700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fce4c02ff60 0x7fce4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.985+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce50041000 con 0x7fce5c0f5410 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.986+0000 7fce5bfff700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fce4c02ff60 0x7fce4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.986+0000 7fce5bfff700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fce4c02ff60 0x7fce4c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fce480096a0 tx=0x7fce48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:02.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:02.990+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce500095f0 con 0x7fce5c0f5410 2024-01-21T20:02:03.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:02.668342+0000 mgr.smithi119.rqacbm (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:03.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.192+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce5c108150 con 0x7fce5c0f5410 2024-01-21T20:02:03.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.193+0000 7fce597fa700 1 -- 172.21.15.153:0/3978135791 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce5001db70 con 0x7fce5c0f5410 2024-01-21T20:02:03.185 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:03.185 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fce4c02ff60 msgr2=0x7fce4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fce4c02ff60 0x7fce4c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fce480096a0 tx=0x7fce48006b40).stop 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 msgr2=0x7fce5c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c106720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fce50013510 tx=0x7fce5000b040).stop 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 shutdown_connections 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fce4c02ff60 0x7fce4c032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:03.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 --2- 172.21.15.153:0/3978135791 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce5c0f5410 0x7fce5c106720 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:03.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 >> 172.21.15.153:0/3978135791 conn(0x7fce5c0f0b70 msgr2=0x7fce5c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:03.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 shutdown_connections 2024-01-21T20:02:03.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:03.195+0000 7fce62b81700 1 -- 172.21.15.153:0/3978135791 wait complete. 2024-01-21T20:02:03.187 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:04.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:03.194608+0000 mon.smithi119 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.153:0/3978135791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:04.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:04.586 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:05.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:04.668699+0000 mgr.smithi119.rqacbm (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:05.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.810+0000 7fa952571700 1 -- 172.21.15.153:0/847399222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 msgr2=0x7fa94c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:05.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.810+0000 7fa952571700 1 --2- 172.21.15.153:0/847399222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c0f8f20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa948009fb0 tx=0x7fa948009b30).stop 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 -- 172.21.15.153:0/847399222 shutdown_connections 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 --2- 172.21.15.153:0/847399222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c0f8f20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 -- 172.21.15.153:0/847399222 >> 172.21.15.153:0/847399222 conn(0x7fa94c0f09a0 msgr2=0x7fa94c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 -- 172.21.15.153:0/847399222 shutdown_connections 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 -- 172.21.15.153:0/847399222 wait complete. 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 Processor -- start 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 -- start start 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:05.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.811+0000 7fa952571700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1069f0 con 0x7fa94c0601e0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa95156f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa95156f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47912/0 (socket says 172.21.15.153:47912) 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa95156f700 1 -- 172.21.15.153:0/2934928518 learned_addr learned my addr 172.21.15.153:0/2934928518 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa95156f700 1 -- 172.21.15.153:0/2934928518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa948005040 con 0x7fa94c0601e0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa95156f700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c106500 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa948013510 tx=0x7fa94800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa948009320 con 0x7fa94c0601e0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c106bf0 con 0x7fa94c0601e0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa948013f00 con 0x7fa94c0601e0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa948014d90 con 0x7fa94c0601e0 2024-01-21T20:02:05.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.812+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c107080 con 0x7fa94c0601e0 2024-01-21T20:02:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.813+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9480095f0 con 0x7fa94c0601e0 2024-01-21T20:02:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.813+0000 7fa941ffb700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa93802ff60 0x7fa938032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.813+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa948041080 con 0x7fa94c0601e0 2024-01-21T20:02:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.813+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c045ad0 con 0x7fa94c0601e0 2024-01-21T20:02:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.813+0000 7fa950d6e700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa93802ff60 0x7fa938032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:05.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.814+0000 7fa950d6e700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa93802ff60 0x7fa938032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa93c0096a0 tx=0x7fa93c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:05.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:05.818+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa948014420 con 0x7fa94c0601e0 2024-01-21T20:02:06.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.023+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa94c107eb0 con 0x7fa94c0601e0 2024-01-21T20:02:06.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.024+0000 7fa941ffb700 1 -- 172.21.15.153:0/2934928518 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa948010230 con 0x7fa94c0601e0 2024-01-21T20:02:06.015 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:06.015 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:06.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa93802ff60 msgr2=0x7fa938032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:06.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa93802ff60 0x7fa938032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa93c0096a0 tx=0x7fa93c006b40).stop 2024-01-21T20:02:06.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 msgr2=0x7fa94c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:06.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c106500 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa948013510 tx=0x7fa94800b040).stop 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 shutdown_connections 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa93802ff60 0x7fa938032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 --2- 172.21.15.153:0/2934928518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa94c0601e0 0x7fa94c106500 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 >> 172.21.15.153:0/2934928518 conn(0x7fa94c0f09a0 msgr2=0x7fa94c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 shutdown_connections 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:06.025+0000 7fa952571700 1 -- 172.21.15.153:0/2934928518 wait complete. 2024-01-21T20:02:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:06.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:06 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:06.025099+0000 mon.smithi119 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.153:0/2934928518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:07.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:07.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:06.669026+0000 mgr.smithi119.rqacbm (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 -- 172.21.15.153:0/683522396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 msgr2=0x7f376c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 --2- 172.21.15.153:0/683522396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c0f8f20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f3754009fb0 tx=0x7f3754009b30).stop 2024-01-21T20:02:08.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 -- 172.21.15.153:0/683522396 shutdown_connections 2024-01-21T20:02:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 --2- 172.21.15.153:0/683522396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c0f8f20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 -- 172.21.15.153:0/683522396 >> 172.21.15.153:0/683522396 conn(0x7f376c0f09a0 msgr2=0x7f376c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 -- 172.21.15.153:0/683522396 shutdown_connections 2024-01-21T20:02:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 -- 172.21.15.153:0/683522396 wait complete. 2024-01-21T20:02:08.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.602+0000 7f377090f700 1 Processor -- start 2024-01-21T20:02:08.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.603+0000 7f377090f700 1 -- start start 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.603+0000 7f377090f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.603+0000 7f377090f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c1069f0 con 0x7f376c0601e0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.603+0000 7f376b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.603+0000 7f376b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47932/0 (socket says 172.21.15.153:47932) 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.603+0000 7f376b7fe700 1 -- 172.21.15.153:0/3079989559 learned_addr learned my addr 172.21.15.153:0/3079989559 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f376b7fe700 1 -- 172.21.15.153:0/3079989559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3754005040 con 0x7f376c0601e0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f376b7fe700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c106500 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3754013510 tx=0x7f375400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3754009320 con 0x7f376c0601e0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376c106bf0 con 0x7f376c0601e0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3754013f00 con 0x7f376c0601e0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3754014d90 con 0x7f376c0601e0 2024-01-21T20:02:08.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.604+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c107080 con 0x7f376c0601e0 2024-01-21T20:02:08.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.605+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3754014420 con 0x7f376c0601e0 2024-01-21T20:02:08.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.605+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376c045ad0 con 0x7f376c0601e0 2024-01-21T20:02:08.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.605+0000 7f3763fff700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f375802ff60 0x7f3758032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:08.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.605+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3754041070 con 0x7f376c0601e0 2024-01-21T20:02:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.605+0000 7f376affd700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f375802ff60 0x7f3758032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:08.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.606+0000 7f376affd700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f375802ff60 0x7f3758032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f375c0096a0 tx=0x7f375c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:08.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.610+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37540095f0 con 0x7f376c0601e0 2024-01-21T20:02:08.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.817+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f376c107f30 con 0x7f376c0601e0 2024-01-21T20:02:08.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.817+0000 7f3763fff700 1 -- 172.21.15.153:0/3079989559 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f375401db70 con 0x7f376c0601e0 2024-01-21T20:02:08.809 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:08.809 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:08.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f375802ff60 msgr2=0x7f3758032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:08.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f375802ff60 0x7f3758032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f375c0096a0 tx=0x7f375c006b40).stop 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 msgr2=0x7f376c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c106500 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3754013510 tx=0x7f375400b040).stop 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 shutdown_connections 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f375802ff60 0x7f3758032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 --2- 172.21.15.153:0/3079989559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f376c0601e0 0x7f376c106500 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 >> 172.21.15.153:0/3079989559 conn(0x7f376c0f09a0 msgr2=0x7f376c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 shutdown_connections 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:08.819+0000 7f377090f700 1 -- 172.21.15.153:0/3079989559 wait complete. 2024-01-21T20:02:08.811 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:09.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:08.669271+0000 mgr.smithi119.rqacbm (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:10.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:08.818960+0000 mon.smithi119 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.153:0/3079989559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:10.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:10.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:10.669626+0000 mgr.smithi119.rqacbm (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:11.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 -- 172.21.15.153:0/3215894516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 msgr2=0x7f79700f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:11.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 --2- 172.21.15.153:0/3215894516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79700f66c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f7964009fb0 tx=0x7f7964009b30).stop 2024-01-21T20:02:11.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 -- 172.21.15.153:0/3215894516 shutdown_connections 2024-01-21T20:02:11.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 --2- 172.21.15.153:0/3215894516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79700f66c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:11.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 -- 172.21.15.153:0/3215894516 >> 172.21.15.153:0/3215894516 conn(0x7f79700f1aa0 msgr2=0x7f79700f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:11.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 -- 172.21.15.153:0/3215894516 shutdown_connections 2024-01-21T20:02:11.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.389+0000 7f796ffff700 1 -- 172.21.15.153:0/3215894516 wait complete. 2024-01-21T20:02:11.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796ffff700 1 Processor -- start 2024-01-21T20:02:11.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796ffff700 1 -- start start 2024-01-21T20:02:11.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79701064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79701069a0 con 0x7f79700f62a0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79701064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79701064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47944/0 (socket says 172.21.15.153:47944) 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.390+0000 7f796effd700 1 -- 172.21.15.153:0/1852371864 learned_addr learned my addr 172.21.15.153:0/1852371864 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f796effd700 1 -- 172.21.15.153:0/1852371864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964005040 con 0x7f79700f62a0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f796effd700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79701064b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7964013510 tx=0x7f796400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7964009320 con 0x7f79700f62a0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7964013f00 con 0x7f79700f62a0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7970106ba0 con 0x7f79700f62a0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7964014dc0 con 0x7f79700f62a0 2024-01-21T20:02:11.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.391+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7970107030 con 0x7f79700f62a0 2024-01-21T20:02:11.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.392+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7964014420 con 0x7f79700f62a0 2024-01-21T20:02:11.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.392+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7970045ad0 con 0x7f79700f62a0 2024-01-21T20:02:11.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.392+0000 7f79577fe700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f795802ff60 0x7f7958032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:11.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.392+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7964041030 con 0x7f79700f62a0 2024-01-21T20:02:11.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.392+0000 7f796e7fc700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f795802ff60 0x7f7958032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:11.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.393+0000 7f796e7fc700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f795802ff60 0x7f7958032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f79600096a0 tx=0x7f7960006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:11.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.397+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79640095f0 con 0x7f79700f62a0 2024-01-21T20:02:11.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.603+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7970107ee0 con 0x7f79700f62a0 2024-01-21T20:02:11.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.604+0000 7f79577fe700 1 -- 172.21.15.153:0/1852371864 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f796401cb10 con 0x7f79700f62a0 2024-01-21T20:02:11.595 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:11.595 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f795802ff60 msgr2=0x7f7958032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f795802ff60 0x7f7958032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f79600096a0 tx=0x7f7960006b40).stop 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 msgr2=0x7f79701064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79701064b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7964013510 tx=0x7f796400b040).stop 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 shutdown_connections 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f795802ff60 0x7f7958032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 --2- 172.21.15.153:0/1852371864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79700f62a0 0x7f79701064b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 >> 172.21.15.153:0/1852371864 conn(0x7f79700f1aa0 msgr2=0x7f79700f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 shutdown_connections 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:11.606+0000 7f796ffff700 1 -- 172.21.15.153:0/1852371864 wait complete. 2024-01-21T20:02:11.598 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:12.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:11.605426+0000 mon.smithi119 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.153:0/1852371864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:13.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:13.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:13.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:12.670005+0000 mgr.smithi119.rqacbm (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 -- 172.21.15.153:0/2854140325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 msgr2=0x7fb2140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 --2- 172.21.15.153:0/2854140325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb2140f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb204009fb0 tx=0x7fb204009b30).stop 2024-01-21T20:02:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 -- 172.21.15.153:0/2854140325 shutdown_connections 2024-01-21T20:02:14.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 --2- 172.21.15.153:0/2854140325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb2140f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 -- 172.21.15.153:0/2854140325 >> 172.21.15.153:0/2854140325 conn(0x7fb2140f0b70 msgr2=0x7fb2140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 -- 172.21.15.153:0/2854140325 shutdown_connections 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.173+0000 7fb21b90e700 1 -- 172.21.15.153:0/2854140325 wait complete. 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb21b90e700 1 Processor -- start 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb21b90e700 1 -- start start 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb21b90e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb214109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb21b90e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214060240 con 0x7fb2140f5410 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb2196aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb214109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb2196aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb214109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44508/0 (socket says 172.21.15.153:44508) 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.174+0000 7fb2196aa700 1 -- 172.21.15.153:0/3280027904 learned_addr learned my addr 172.21.15.153:0/3280027904 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb2196aa700 1 -- 172.21.15.153:0/3280027904 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204005040 con 0x7fb2140f5410 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb2196aa700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb214109430 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb204013510 tx=0x7fb20400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb204009320 con 0x7fb2140f5410 2024-01-21T20:02:14.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb204013f00 con 0x7fb2140f5410 2024-01-21T20:02:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb214060440 con 0x7fb2140f5410 2024-01-21T20:02:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb204014d90 con 0x7fb2140f5410 2024-01-21T20:02:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.175+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb214109ae0 con 0x7fb2140f5410 2024-01-21T20:02:14.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.176+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb214045ad0 con 0x7fb2140f5410 2024-01-21T20:02:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.176+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2040095f0 con 0x7fb2140f5410 2024-01-21T20:02:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.176+0000 7fb209ffb700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb20002ff60 0x7fb200032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.176+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb204040fc0 con 0x7fb2140f5410 2024-01-21T20:02:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.176+0000 7fb218ea9700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb20002ff60 0x7fb200032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:14.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.177+0000 7fb218ea9700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb20002ff60 0x7fb200032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb2100096a0 tx=0x7fb210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:14.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.181+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb204014420 con 0x7fb2140f5410 2024-01-21T20:02:14.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.388+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2140fae60 con 0x7fb2140f5410 2024-01-21T20:02:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.389+0000 7fb209ffb700 1 -- 172.21.15.153:0/3280027904 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb204010230 con 0x7fb2140f5410 2024-01-21T20:02:14.380 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:14.380 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb20002ff60 msgr2=0x7fb200032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb20002ff60 0x7fb200032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb2100096a0 tx=0x7fb210006b40).stop 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 msgr2=0x7fb214109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb214109430 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb204013510 tx=0x7fb20400b040).stop 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 shutdown_connections 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb20002ff60 0x7fb200032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 --2- 172.21.15.153:0/3280027904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2140f5410 0x7fb214109430 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.390+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 >> 172.21.15.153:0/3280027904 conn(0x7fb2140f0b70 msgr2=0x7fb2140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.391+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 shutdown_connections 2024-01-21T20:02:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:14.391+0000 7fb21b90e700 1 -- 172.21.15.153:0/3280027904 wait complete. 2024-01-21T20:02:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:14.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:14.390179+0000 mon.smithi119 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.153:0/3280027904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:15.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:15.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:15.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:15 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:14.670277+0000 mgr.smithi119.rqacbm (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:16.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 -- 172.21.15.153:0/3660630988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 msgr2=0x7f914c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 --2- 172.21.15.153:0/3660630988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c0f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f913c009fb0 tx=0x7f913c009b30).stop 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 -- 172.21.15.153:0/3660630988 shutdown_connections 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 --2- 172.21.15.153:0/3660630988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c0f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 -- 172.21.15.153:0/3660630988 >> 172.21.15.153:0/3660630988 conn(0x7f914c0f0b70 msgr2=0x7f914c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 -- 172.21.15.153:0/3660630988 shutdown_connections 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.963+0000 7f915277e700 1 -- 172.21.15.153:0/3660630988 wait complete. 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.964+0000 7f915277e700 1 Processor -- start 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.964+0000 7f915277e700 1 -- start start 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.964+0000 7f915277e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.964+0000 7f915277e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c060220 con 0x7f914c0f5410 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f914bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f914bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44526/0 (socket says 172.21.15.153:44526) 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f914bfff700 1 -- 172.21.15.153:0/3465291643 learned_addr learned my addr 172.21.15.153:0/3465291643 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:16.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f914bfff700 1 -- 172.21.15.153:0/3465291643 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c005040 con 0x7f914c0f5410 2024-01-21T20:02:16.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f914bfff700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c109450 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f913c013510 tx=0x7f913c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:16.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c009320 con 0x7f914c0f5410 2024-01-21T20:02:16.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c060420 con 0x7f914c0f5410 2024-01-21T20:02:16.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f913c013f00 con 0x7f914c0f5410 2024-01-21T20:02:16.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c014d90 con 0x7f914c0f5410 2024-01-21T20:02:16.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.965+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c109b00 con 0x7f914c0f5410 2024-01-21T20:02:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.966+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f913c0095f0 con 0x7f914c0f5410 2024-01-21T20:02:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.966+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c059100 con 0x7f914c0f5410 2024-01-21T20:02:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.966+0000 7f9148ff9700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f913402ffb0 0x7f9134032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.966+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f913c040fe0 con 0x7f914c0f5410 2024-01-21T20:02:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.967+0000 7f914b7fe700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f913402ffb0 0x7f9134032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:16.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.967+0000 7f914b7fe700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f913402ffb0 0x7f9134032470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f91400096a0 tx=0x7f9140006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:16.971+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f913c06d2b0 con 0x7f914c0f5410 2024-01-21T20:02:17.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.175+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f914c045ad0 con 0x7f914c0f5410 2024-01-21T20:02:17.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.176+0000 7f9148ff9700 1 -- 172.21.15.153:0/3465291643 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f913c014420 con 0x7f914c0f5410 2024-01-21T20:02:17.167 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:17.167 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f913402ffb0 msgr2=0x7f9134032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f913402ffb0 0x7f9134032470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f91400096a0 tx=0x7f9140006b40).stop 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 msgr2=0x7f914c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c109450 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f913c013510 tx=0x7f913c00b040).stop 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 shutdown_connections 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f913402ffb0 0x7f9134032470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 --2- 172.21.15.153:0/3465291643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0f5410 0x7f914c109450 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 >> 172.21.15.153:0/3465291643 conn(0x7f914c0f0b70 msgr2=0x7f914c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 shutdown_connections 2024-01-21T20:02:17.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:17.177+0000 7f915277e700 1 -- 172.21.15.153:0/3465291643 wait complete. 2024-01-21T20:02:17.170 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:16.670623+0000 mgr.smithi119.rqacbm (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:18.563 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:18.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:18.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:17.177260+0000 mon.smithi119 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.153:0/3465291643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:19.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:18.670862+0000 mgr.smithi119.rqacbm (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/2996140564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 msgr2=0x7fa8a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 --2- 172.21.15.153:0/2996140564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a00f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fa894009fb0 tx=0x7fa894009b30).stop 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/2996140564 shutdown_connections 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 --2- 172.21.15.153:0/2996140564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a00f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/2996140564 >> 172.21.15.153:0/2996140564 conn(0x7fa8a00f1c70 msgr2=0x7fa8a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/2996140564 shutdown_connections 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/2996140564 wait complete. 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.755+0000 7fa8a4cdf700 1 Processor -- start 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa8a4cdf700 1 -- start start 2024-01-21T20:02:19.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa8a4cdf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa8a4cdf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0069e20 con 0x7fa8a00f6510 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa89e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa89e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44540/0 (socket says 172.21.15.153:44540) 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa89e59c700 1 -- 172.21.15.153:0/3536423139 learned_addr learned my addr 172.21.15.153:0/3536423139 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa89e59c700 1 -- 172.21.15.153:0/3536423139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa894005040 con 0x7fa8a00f6510 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa89e59c700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a006b9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa894013510 tx=0x7fa89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa894009320 con 0x7fa8a00f6510 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a0069fc0 con 0x7fa8a00f6510 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa894013f00 con 0x7fa8a00f6510 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.756+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa894014d90 con 0x7fa8a00f6510 2024-01-21T20:02:19.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.757+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a006a450 con 0x7fa8a00f6510 2024-01-21T20:02:19.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.757+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8940095f0 con 0x7fa8a00f6510 2024-01-21T20:02:19.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.757+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a0059100 con 0x7fa8a00f6510 2024-01-21T20:02:19.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.757+0000 7fa88effd700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa88802ff60 0x7fa888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:19.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.757+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa894041080 con 0x7fa8a00f6510 2024-01-21T20:02:19.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.758+0000 7fa89dd9b700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa88802ff60 0x7fa888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:19.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.758+0000 7fa89dd9b700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa88802ff60 0x7fa888032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa8900096a0 tx=0x7fa890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:19.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.762+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa894014420 con 0x7fa8a00f6510 2024-01-21T20:02:19.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.967+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8a0045ad0 con 0x7fa8a00f6510 2024-01-21T20:02:19.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.967+0000 7fa88effd700 1 -- 172.21.15.153:0/3536423139 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa894010230 con 0x7fa8a00f6510 2024-01-21T20:02:19.959 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:19.959 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:19.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa88802ff60 msgr2=0x7fa888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa88802ff60 0x7fa888032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa8900096a0 tx=0x7fa890006b40).stop 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 msgr2=0x7fa8a006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a006b9a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fa894013510 tx=0x7fa89400b040).stop 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 shutdown_connections 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa88802ff60 0x7fa888032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 --2- 172.21.15.153:0/3536423139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8a00f6510 0x7fa8a006b9a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 >> 172.21.15.153:0/3536423139 conn(0x7fa8a00f1c70 msgr2=0x7fa8a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 shutdown_connections 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:19.969+0000 7fa8a4cdf700 1 -- 172.21.15.153:0/3536423139 wait complete. 2024-01-21T20:02:19.961 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:20.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:19.968929+0000 mon.smithi119 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.153:0/3536423139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:21.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:21.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:21.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:20.671200+0000 mgr.smithi119.rqacbm (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.584+0000 7fe31e660700 1 -- 172.21.15.153:0/1718129477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 msgr2=0x7fe3180f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.584+0000 7fe31e660700 1 --2- 172.21.15.153:0/1718129477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3180f68a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fe308009fb0 tx=0x7fe308009b30).stop 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 -- 172.21.15.153:0/1718129477 shutdown_connections 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 --2- 172.21.15.153:0/1718129477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3180f68a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 -- 172.21.15.153:0/1718129477 >> 172.21.15.153:0/1718129477 conn(0x7fe3180f1c00 msgr2=0x7fe3180f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 -- 172.21.15.153:0/1718129477 shutdown_connections 2024-01-21T20:02:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 -- 172.21.15.153:0/1718129477 wait complete. 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 Processor -- start 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 -- start start 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3181066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.585+0000 7fe31e660700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318106bc0 con 0x7fe3180f6480 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe317fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3181066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe317fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3181066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44554/0 (socket says 172.21.15.153:44554) 2024-01-21T20:02:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe317fff700 1 -- 172.21.15.153:0/2713006987 learned_addr learned my addr 172.21.15.153:0/2713006987 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe317fff700 1 -- 172.21.15.153:0/2713006987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe308005040 con 0x7fe3180f6480 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe317fff700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3181066d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe308013510 tx=0x7fe30800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe308009320 con 0x7fe3180f6480 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe318106dc0 con 0x7fe3180f6480 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe308013f00 con 0x7fe3180f6480 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe308014d90 con 0x7fe3180f6480 2024-01-21T20:02:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.586+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe318107250 con 0x7fe3180f6480 2024-01-21T20:02:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.587+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3080263a0 con 0x7fe3180f6480 2024-01-21T20:02:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.587+0000 7fe314ff9700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe30002ff60 0x7fe300032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.587+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe318045ad0 con 0x7fe3180f6480 2024-01-21T20:02:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.587+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3080410c0 con 0x7fe3180f6480 2024-01-21T20:02:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.587+0000 7fe3177fe700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe30002ff60 0x7fe300032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.588+0000 7fe3177fe700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe30002ff60 0x7fe300032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe30c0096a0 tx=0x7fe30c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.592+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3080095f0 con 0x7fe3180f6480 2024-01-21T20:02:22.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.795+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe318108100 con 0x7fe3180f6480 2024-01-21T20:02:22.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.796+0000 7fe314ff9700 1 -- 172.21.15.153:0/2713006987 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe308040030 con 0x7fe3180f6480 2024-01-21T20:02:22.788 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:22.788 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe30002ff60 msgr2=0x7fe300032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe30002ff60 0x7fe300032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fe30c0096a0 tx=0x7fe30c006b40).stop 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 msgr2=0x7fe3181066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3181066d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe308013510 tx=0x7fe30800b040).stop 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 shutdown_connections 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe30002ff60 0x7fe300032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:22.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 --2- 172.21.15.153:0/2713006987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3180f6480 0x7fe3181066d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:22.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 >> 172.21.15.153:0/2713006987 conn(0x7fe3180f1c00 msgr2=0x7fe3180f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:22.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 shutdown_connections 2024-01-21T20:02:22.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:22.798+0000 7fe31e660700 1 -- 172.21.15.153:0/2713006987 wait complete. 2024-01-21T20:02:22.790 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:23.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:22.671533+0000 mgr.smithi119.rqacbm (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:24.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:22.797671+0000 mon.smithi119 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.153:0/2713006987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:24.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:24.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:25.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:24.671791+0000 mgr.smithi119.rqacbm (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:25.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- 172.21.15.153:0/428921615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 msgr2=0x7f55900f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:25.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 --2- 172.21.15.153:0/428921615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f55900f4b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f558c009fb0 tx=0x7f558c009b30).stop 2024-01-21T20:02:25.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- 172.21.15.153:0/428921615 shutdown_connections 2024-01-21T20:02:25.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 --2- 172.21.15.153:0/428921615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f55900f4b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:25.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- 172.21.15.153:0/428921615 >> 172.21.15.153:0/428921615 conn(0x7f55900f0120 msgr2=0x7f55900f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:25.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- 172.21.15.153:0/428921615 shutdown_connections 2024-01-21T20:02:25.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- 172.21.15.153:0/428921615 wait complete. 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 Processor -- start 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- start start 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f5590106510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.390+0000 7f5596336700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590106a00 con 0x7f55900f46e0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5595334700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f5590106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5595334700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f5590106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47402/0 (socket says 172.21.15.153:47402) 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5595334700 1 -- 172.21.15.153:0/3264206254 learned_addr learned my addr 172.21.15.153:0/3264206254 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5595334700 1 -- 172.21.15.153:0/3264206254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c005040 con 0x7f55900f46e0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5595334700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f5590106510 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f558c013510 tx=0x7f558c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558c009320 con 0x7f55900f46e0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5590106c00 con 0x7f55900f46e0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f558c013f00 con 0x7f55900f46e0 2024-01-21T20:02:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.391+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558c014dc0 con 0x7f55900f46e0 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.392+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5590107090 con 0x7f55900f46e0 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.392+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f558c0095f0 con 0x7f55900f46e0 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.392+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5590045ad0 con 0x7f55900f46e0 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.392+0000 7f5585ffb700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f557c02ff60 0x7f557c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.392+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f558c0410c0 con 0x7f55900f46e0 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.393+0000 7f5594b33700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f557c02ff60 0x7f557c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.393+0000 7f5594b33700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f557c02ff60 0x7f557c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f55800096a0 tx=0x7f5580006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.397+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f558c014420 con 0x7f55900f46e0 2024-01-21T20:02:25.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.607+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5590107f40 con 0x7f55900f46e0 2024-01-21T20:02:25.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.608+0000 7f5585ffb700 1 -- 172.21.15.153:0/3264206254 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f558c010230 con 0x7f55900f46e0 2024-01-21T20:02:25.600 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:25.600 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f557c02ff60 msgr2=0x7f557c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f557c02ff60 0x7f557c032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f55800096a0 tx=0x7f5580006b40).stop 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 msgr2=0x7f5590106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f5590106510 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f558c013510 tx=0x7f558c00b040).stop 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 shutdown_connections 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f557c02ff60 0x7f557c032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 --2- 172.21.15.153:0/3264206254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55900f46e0 0x7f5590106510 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:25.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 >> 172.21.15.153:0/3264206254 conn(0x7f55900f0120 msgr2=0x7f55900f0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:25.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 shutdown_connections 2024-01-21T20:02:25.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:25.610+0000 7f5596336700 1 -- 172.21.15.153:0/3264206254 wait complete. 2024-01-21T20:02:25.602 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:26.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:25.609437+0000 mon.smithi119 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.153:0/3264206254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:26.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:26.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:26.672020+0000 mgr.smithi119.rqacbm (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 -- 172.21.15.153:0/443845917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 msgr2=0x7f0f200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 --2- 172.21.15.153:0/443845917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f200f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0f14009fb0 tx=0x7f0f14009b30).stop 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 -- 172.21.15.153:0/443845917 shutdown_connections 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 --2- 172.21.15.153:0/443845917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f200f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 -- 172.21.15.153:0/443845917 >> 172.21.15.153:0/443845917 conn(0x7f0f200f0b70 msgr2=0x7f0f200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 -- 172.21.15.153:0/443845917 shutdown_connections 2024-01-21T20:02:28.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.116+0000 7f0f26da6700 1 -- 172.21.15.153:0/443845917 wait complete. 2024-01-21T20:02:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f26da6700 1 Processor -- start 2024-01-21T20:02:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f26da6700 1 -- start start 2024-01-21T20:02:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f26da6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f20105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f26da6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f20105af0 con 0x7f0f200f5410 2024-01-21T20:02:28.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f24b42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f20105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f24b42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f20105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47420/0 (socket says 172.21.15.153:47420) 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.117+0000 7f0f24b42700 1 -- 172.21.15.153:0/1162348007 learned_addr learned my addr 172.21.15.153:0/1162348007 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f24b42700 1 -- 172.21.15.153:0/1162348007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14005040 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f24b42700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f20105600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0f14013510 tx=0x7f0f1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f14009320 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f20105cf0 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f14013f00 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f14014d90 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.118+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f20106180 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.119+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f140263a0 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.119+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f20045ad0 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.119+0000 7f0f1d7fa700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f1002ff60 0x7f0f10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.119+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f14041390 con 0x7f0f200f5410 2024-01-21T20:02:28.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.119+0000 7f0f1ffff700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f1002ff60 0x7f0f10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:28.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.119+0000 7f0f1ffff700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f1002ff60 0x7f0f10032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0f0c0096a0 tx=0x7f0f0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.124+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f1401cb10 con 0x7f0f200f5410 2024-01-21T20:02:28.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.329+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f200fae60 con 0x7f0f200f5410 2024-01-21T20:02:28.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.330+0000 7f0f1d7fa700 1 -- 172.21.15.153:0/1162348007 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f1401cb10 con 0x7f0f200f5410 2024-01-21T20:02:28.321 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:28.321 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f1002ff60 msgr2=0x7f0f10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:28.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f1002ff60 0x7f0f10032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0f0c0096a0 tx=0x7f0f0c006b40).stop 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 msgr2=0x7f0f20105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f20105600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0f14013510 tx=0x7f0f1400b040).stop 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 shutdown_connections 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f1002ff60 0x7f0f10032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 --2- 172.21.15.153:0/1162348007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f200f5410 0x7f0f20105600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 >> 172.21.15.153:0/1162348007 conn(0x7f0f200f0b70 msgr2=0x7f0f200f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 shutdown_connections 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:28.331+0000 7f0f26da6700 1 -- 172.21.15.153:0/1162348007 wait complete. 2024-01-21T20:02:28.323 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:28.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:28.331186+0000 mon.smithi119 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.153:0/1162348007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:29.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:29.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:29.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:28.672273+0000 mgr.smithi119.rqacbm (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:30.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 -- 172.21.15.153:0/4097202102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 msgr2=0x7fe3000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:30.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 --2- 172.21.15.153:0/4097202102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe3000f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe2f0009fb0 tx=0x7fe2f0009b30).stop 2024-01-21T20:02:30.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 -- 172.21.15.153:0/4097202102 shutdown_connections 2024-01-21T20:02:30.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 --2- 172.21.15.153:0/4097202102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe3000f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:30.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 -- 172.21.15.153:0/4097202102 >> 172.21.15.153:0/4097202102 conn(0x7fe3000f0b70 msgr2=0x7fe3000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 -- 172.21.15.153:0/4097202102 shutdown_connections 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.922+0000 7fe305dd3700 1 -- 172.21.15.153:0/4097202102 wait complete. 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe305dd3700 1 Processor -- start 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe305dd3700 1 -- start start 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe305dd3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe300108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe305dd3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300108e00 con 0x7fe3000f5410 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe2ff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe300108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe2ff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe300108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47444/0 (socket says 172.21.15.153:47444) 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.923+0000 7fe2ff7fe700 1 -- 172.21.15.153:0/2866965440 learned_addr learned my addr 172.21.15.153:0/2866965440 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe2ff7fe700 1 -- 172.21.15.153:0/2866965440 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0005040 con 0x7fe3000f5410 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe2ff7fe700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe300108910 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fe2f0013510 tx=0x7fe2f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f0009320 con 0x7fe3000f5410 2024-01-21T20:02:30.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2f0013f00 con 0x7fe3000f5410 2024-01-21T20:02:30.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe300109000 con 0x7fe3000f5410 2024-01-21T20:02:30.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f0014d90 con 0x7fe3000f5410 2024-01-21T20:02:30.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.924+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe300109490 con 0x7fe3000f5410 2024-01-21T20:02:30.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.925+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2f00263a0 con 0x7fe3000f5410 2024-01-21T20:02:30.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.925+0000 7fe2e7fff700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2e802ff60 0x7fe2e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:30.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.925+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2f00410c0 con 0x7fe3000f5410 2024-01-21T20:02:30.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.925+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300045ad0 con 0x7fe3000f5410 2024-01-21T20:02:30.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.925+0000 7fe2feffd700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2e802ff60 0x7fe2e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:30.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.926+0000 7fe2feffd700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2e802ff60 0x7fe2e8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe2f40096a0 tx=0x7fe2f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:30.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:30.930+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2f0012920 con 0x7fe3000f5410 2024-01-21T20:02:31.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:30.672517+0000 mgr.smithi119.rqacbm (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:31.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.134+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3000fae60 con 0x7fe3000f5410 2024-01-21T20:02:31.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.135+0000 7fe2e7fff700 1 -- 172.21.15.153:0/2866965440 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe2f002b020 con 0x7fe3000f5410 2024-01-21T20:02:31.126 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:31.126 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:31.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2e802ff60 msgr2=0x7fe2e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:31.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2e802ff60 0x7fe2e8032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe2f40096a0 tx=0x7fe2f4006b40).stop 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 msgr2=0x7fe300108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe300108910 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fe2f0013510 tx=0x7fe2f000b040).stop 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 shutdown_connections 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe2e802ff60 0x7fe2e8032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 --2- 172.21.15.153:0/2866965440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3000f5410 0x7fe300108910 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 >> 172.21.15.153:0/2866965440 conn(0x7fe3000f0b70 msgr2=0x7fe30005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 shutdown_connections 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:31.136+0000 7fe305dd3700 1 -- 172.21.15.153:0/2866965440 wait complete. 2024-01-21T20:02:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:32.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:32.520 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:32.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:31.136093+0000 mon.smithi119 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.153:0/2866965440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:33.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:32.672820+0000 mgr.smithi119.rqacbm (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:33.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 -- 172.21.15.153:0/1181399914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 msgr2=0x7fae200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 --2- 172.21.15.153:0/1181399914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae200f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fae10009fb0 tx=0x7fae10009b30).stop 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 -- 172.21.15.153:0/1181399914 shutdown_connections 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 --2- 172.21.15.153:0/1181399914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae200f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 -- 172.21.15.153:0/1181399914 >> 172.21.15.153:0/1181399914 conn(0x7fae200f0b70 msgr2=0x7fae200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 -- 172.21.15.153:0/1181399914 shutdown_connections 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 -- 172.21.15.153:0/1181399914 wait complete. 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.699+0000 7fae24b17700 1 Processor -- start 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae24b17700 1 -- start start 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae24b17700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae20109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae24b17700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20060240 con 0x7fae200f5410 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae1e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae20109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:33.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae1e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae20109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41714/0 (socket says 172.21.15.153:41714) 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae1e59c700 1 -- 172.21.15.153:0/4039691493 learned_addr learned my addr 172.21.15.153:0/4039691493 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae1e59c700 1 -- 172.21.15.153:0/4039691493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae10005040 con 0x7fae200f5410 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae1e59c700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae20109400 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fae10014000 tx=0x7fae1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae100095f0 con 0x7fae200f5410 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae20060440 con 0x7fae200f5410 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae10007e70 con 0x7fae200f5410 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.700+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae10014ea0 con 0x7fae200f5410 2024-01-21T20:02:33.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.701+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae20109ab0 con 0x7fae200f5410 2024-01-21T20:02:33.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.701+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae10025070 con 0x7fae200f5410 2024-01-21T20:02:33.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.701+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae20045ad0 con 0x7fae200f5410 2024-01-21T20:02:33.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.702+0000 7fae0effd700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fae0802ffb0 0x7fae08032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:33.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.702+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae10040a30 con 0x7fae200f5410 2024-01-21T20:02:33.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.702+0000 7fae1dd9b700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fae0802ffb0 0x7fae08032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:33.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.702+0000 7fae1dd9b700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fae0802ffb0 0x7fae08032470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:33.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.706+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae100079c0 con 0x7fae200f5410 2024-01-21T20:02:33.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.911+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae200fae60 con 0x7fae200f5410 2024-01-21T20:02:33.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.912+0000 7fae0effd700 1 -- 172.21.15.153:0/4039691493 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fae1000f300 con 0x7fae200f5410 2024-01-21T20:02:33.902 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:33.902 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fae0802ffb0 msgr2=0x7fae08032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fae0802ffb0 0x7fae08032470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).stop 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 msgr2=0x7fae20109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae20109400 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fae10014000 tx=0x7fae1000b040).stop 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 shutdown_connections 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fae0802ffb0 0x7fae08032470 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 --2- 172.21.15.153:0/4039691493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae200f5410 0x7fae20109400 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:33.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 >> 172.21.15.153:0/4039691493 conn(0x7fae200f0b70 msgr2=0x7fae200f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:33.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 shutdown_connections 2024-01-21T20:02:33.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:33.914+0000 7fae24b17700 1 -- 172.21.15.153:0/4039691493 wait complete. 2024-01-21T20:02:33.905 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:34.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:33.912281+0000 mon.smithi119 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.153:0/4039691493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:35.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:35.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:35.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:34.673036+0000 mgr.smithi119.rqacbm (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:36.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 -- 172.21.15.153:0/3087995108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 msgr2=0x7efc940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:36.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 --2- 172.21.15.153:0/3087995108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc940f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7efc84009fb0 tx=0x7efc84009b30).stop 2024-01-21T20:02:36.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 -- 172.21.15.153:0/3087995108 shutdown_connections 2024-01-21T20:02:36.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 --2- 172.21.15.153:0/3087995108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc940f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:36.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 -- 172.21.15.153:0/3087995108 >> 172.21.15.153:0/3087995108 conn(0x7efc940f1c70 msgr2=0x7efc940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:36.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 -- 172.21.15.153:0/3087995108 shutdown_connections 2024-01-21T20:02:36.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.462+0000 7efc9b2a6700 1 -- 172.21.15.153:0/3087995108 wait complete. 2024-01-21T20:02:36.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc9b2a6700 1 Processor -- start 2024-01-21T20:02:36.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc9b2a6700 1 -- start start 2024-01-21T20:02:36.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc9b2a6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc94106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:36.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc9b2a6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94106c20 con 0x7efc940f6510 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc99042700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc94106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc99042700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc94106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41740/0 (socket says 172.21.15.153:41740) 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.463+0000 7efc99042700 1 -- 172.21.15.153:0/2540358473 learned_addr learned my addr 172.21.15.153:0/2540358473 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc99042700 1 -- 172.21.15.153:0/2540358473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84005040 con 0x7efc940f6510 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc99042700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc94106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7efc84013510 tx=0x7efc8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc84009320 con 0x7efc940f6510 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc84013f00 con 0x7efc940f6510 2024-01-21T20:02:36.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc94106e20 con 0x7efc940f6510 2024-01-21T20:02:36.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc84014d90 con 0x7efc940f6510 2024-01-21T20:02:36.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.464+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc941072b0 con 0x7efc940f6510 2024-01-21T20:02:36.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.465+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc84014420 con 0x7efc940f6510 2024-01-21T20:02:36.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.465+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc94045ad0 con 0x7efc940f6510 2024-01-21T20:02:36.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.465+0000 7efc89ffb700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc8002ff60 0x7efc80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:36.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.465+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc84041000 con 0x7efc940f6510 2024-01-21T20:02:36.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.466+0000 7efc98841700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc8002ff60 0x7efc80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:36.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.466+0000 7efc98841700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc8002ff60 0x7efc80032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7efc900096a0 tx=0x7efc90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:36.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.470+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc84026bf0 con 0x7efc940f6510 2024-01-21T20:02:36.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.676+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc940fbf60 con 0x7efc940f6510 2024-01-21T20:02:36.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.676+0000 7efc89ffb700 1 -- 172.21.15.153:0/2540358473 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efc8401db70 con 0x7efc940f6510 2024-01-21T20:02:36.667 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:36.667 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:36.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc8002ff60 msgr2=0x7efc80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:36.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc8002ff60 0x7efc80032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7efc900096a0 tx=0x7efc90006b40).stop 2024-01-21T20:02:36.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 msgr2=0x7efc94106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:36.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc94106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7efc84013510 tx=0x7efc8400b040).stop 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 shutdown_connections 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc8002ff60 0x7efc80032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 --2- 172.21.15.153:0/2540358473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc940f6510 0x7efc94106730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 >> 172.21.15.153:0/2540358473 conn(0x7efc940f1c70 msgr2=0x7efc940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 shutdown_connections 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:36.678+0000 7efc9b2a6700 1 -- 172.21.15.153:0/2540358473 wait complete. 2024-01-21T20:02:36.669 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:36.673393+0000 mgr.smithi119.rqacbm (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:37.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:36.676851+0000 mon.smithi119 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.153:0/2540358473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:38.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:38.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:39.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:38.673687+0000 mgr.smithi119.rqacbm (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 -- 172.21.15.153:0/1570398935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 msgr2=0x7fa9200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 --2- 172.21.15.153:0/1570398935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa9200f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa908009fb0 tx=0x7fa908009b30).stop 2024-01-21T20:02:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 -- 172.21.15.153:0/1570398935 shutdown_connections 2024-01-21T20:02:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 --2- 172.21.15.153:0/1570398935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa9200f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 -- 172.21.15.153:0/1570398935 >> 172.21.15.153:0/1570398935 conn(0x7fa9200f1c70 msgr2=0x7fa9200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:39.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 -- 172.21.15.153:0/1570398935 shutdown_connections 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.269+0000 7fa924b44700 1 -- 172.21.15.153:0/1570398935 wait complete. 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa924b44700 1 Processor -- start 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa924b44700 1 -- start start 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa924b44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa924b44700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920069e20 con 0x7fa9200f6510 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa91e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa91e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41746/0 (socket says 172.21.15.153:41746) 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.270+0000 7fa91e59c700 1 -- 172.21.15.153:0/4126029511 learned_addr learned my addr 172.21.15.153:0/4126029511 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa91e59c700 1 -- 172.21.15.153:0/4126029511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908005040 con 0x7fa9200f6510 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa91e59c700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa908013510 tx=0x7fa90800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908009320 con 0x7fa9200f6510 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa908013f00 con 0x7fa9200f6510 2024-01-21T20:02:39.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa920069fc0 con 0x7fa9200f6510 2024-01-21T20:02:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa908014d90 con 0x7fa9200f6510 2024-01-21T20:02:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.271+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92006a450 con 0x7fa9200f6510 2024-01-21T20:02:39.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.272+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9080263a0 con 0x7fa9200f6510 2024-01-21T20:02:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.272+0000 7fa916ffd700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa90c02ff60 0x7fa90c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.272+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9080410e0 con 0x7fa9200f6510 2024-01-21T20:02:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.272+0000 7fa91dd9b700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa90c02ff60 0x7fa90c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.273+0000 7fa91dd9b700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa90c02ff60 0x7fa90c032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa9100096a0 tx=0x7fa910006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:39.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.274+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa920059100 con 0x7fa9200f6510 2024-01-21T20:02:39.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.279+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa908012920 con 0x7fa9200f6510 2024-01-21T20:02:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.486+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa92006aab0 con 0x7fa9200f6510 2024-01-21T20:02:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.486+0000 7fa916ffd700 1 -- 172.21.15.153:0/4126029511 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa90802b020 con 0x7fa9200f6510 2024-01-21T20:02:39.477 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:39.477 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa90c02ff60 msgr2=0x7fa90c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa90c02ff60 0x7fa90c032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fa9100096a0 tx=0x7fa910006b40).stop 2024-01-21T20:02:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 msgr2=0x7fa92006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa908013510 tx=0x7fa90800b040).stop 2024-01-21T20:02:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 shutdown_connections 2024-01-21T20:02:39.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa90c02ff60 0x7fa90c032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 --2- 172.21.15.153:0/4126029511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9200f6510 0x7fa92006b9a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 >> 172.21.15.153:0/4126029511 conn(0x7fa9200f1c70 msgr2=0x7fa9200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 shutdown_connections 2024-01-21T20:02:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:39.488+0000 7fa924b44700 1 -- 172.21.15.153:0/4126029511 wait complete. 2024-01-21T20:02:39.479 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:40.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:39.486724+0000 mon.smithi119 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.153:0/4126029511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:40.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:40.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:41.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:40.673929+0000 mgr.smithi119.rqacbm (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 -- 172.21.15.153:0/1754135224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 msgr2=0x7f51400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 --2- 172.21.15.153:0/1754135224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5134009fb0 tx=0x7f5134009b30).stop 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 -- 172.21.15.153:0/1754135224 shutdown_connections 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 --2- 172.21.15.153:0/1754135224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 -- 172.21.15.153:0/1754135224 >> 172.21.15.153:0/1754135224 conn(0x7f51400f0b70 msgr2=0x7f51400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 -- 172.21.15.153:0/1754135224 shutdown_connections 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.064+0000 7f5146d8c700 1 -- 172.21.15.153:0/1754135224 wait complete. 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5146d8c700 1 Processor -- start 2024-01-21T20:02:42.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5146d8c700 1 -- start start 2024-01-21T20:02:42.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5146d8c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:42.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5146d8c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51400fb820 con 0x7f51400f5410 2024-01-21T20:02:42.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5144b28700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:42.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5144b28700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:41764/0 (socket says 172.21.15.153:41764) 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5144b28700 1 -- 172.21.15.153:0/3420868740 learned_addr learned my addr 172.21.15.153:0/3420868740 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.065+0000 7f5144b28700 1 -- 172.21.15.153:0/3420868740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5134005040 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.066+0000 7f5144b28700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5134013510 tx=0x7f513400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.066+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5134009320 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.066+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51400fb9c0 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.066+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5134013f00 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.066+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5134014d90 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.066+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51400fbe50 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.067+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51340095f0 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.067+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5140059100 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.067+0000 7f513d7fa700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f513002ff60 0x7f5130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.067+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51340410c0 con 0x7f51400f5410 2024-01-21T20:02:42.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.067+0000 7f513ffff700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f513002ff60 0x7f5130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:42.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.067+0000 7f513ffff700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f513002ff60 0x7f5130032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f512c0096a0 tx=0x7f512c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:42.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.071+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5134014420 con 0x7f51400f5410 2024-01-21T20:02:42.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.278+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51400fae60 con 0x7f51400f5410 2024-01-21T20:02:42.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.278+0000 7f513d7fa700 1 -- 172.21.15.153:0/3420868740 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f513401d7f0 con 0x7f51400f5410 2024-01-21T20:02:42.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:42.269 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:42.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f513002ff60 msgr2=0x7f5130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:42.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f513002ff60 0x7f5130032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f512c0096a0 tx=0x7f512c006b40).stop 2024-01-21T20:02:42.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 msgr2=0x7f51400fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:42.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5134013510 tx=0x7f513400b040).stop 2024-01-21T20:02:42.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 shutdown_connections 2024-01-21T20:02:42.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f513002ff60 0x7f5130032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:42.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 --2- 172.21.15.153:0/3420868740 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51400f5410 0x7f51400fd3a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:42.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 >> 172.21.15.153:0/3420868740 conn(0x7f51400f0b70 msgr2=0x7f514005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:42.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 shutdown_connections 2024-01-21T20:02:42.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:42.280+0000 7f5146d8c700 1 -- 172.21.15.153:0/3420868740 wait complete. 2024-01-21T20:02:42.271 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:42.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:42.278889+0000 mon.smithi119 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.153:0/3420868740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:43.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:42.674202+0000 mgr.smithi119.rqacbm (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:43.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:43.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 -- 172.21.15.153:0/3745052755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 msgr2=0x7f96140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 --2- 172.21.15.153:0/3745052755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96140f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f9604009fb0 tx=0x7f9604009b30).stop 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 -- 172.21.15.153:0/3745052755 shutdown_connections 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 --2- 172.21.15.153:0/3745052755 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96140f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 -- 172.21.15.153:0/3745052755 >> 172.21.15.153:0/3745052755 conn(0x7f96140f09a0 msgr2=0x7f96140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 -- 172.21.15.153:0/3745052755 shutdown_connections 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 -- 172.21.15.153:0/3745052755 wait complete. 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 Processor -- start 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.864+0000 7f961359e700 1 -- start start 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96141064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:44.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96141069d0 con 0x7f96140f51a0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96141064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96141064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47988/0 (socket says 172.21.15.153:47988) 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961259c700 1 -- 172.21.15.153:0/622629782 learned_addr learned my addr 172.21.15.153:0/622629782 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961259c700 1 -- 172.21.15.153:0/622629782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9604005040 con 0x7f96140f51a0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961259c700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96141064e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9604013510 tx=0x7f960400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9604009320 con 0x7f96140f51a0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614106bd0 con 0x7f96140f51a0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9604013f00 con 0x7f96140f51a0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9604014dc0 con 0x7f96140f51a0 2024-01-21T20:02:44.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.865+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9614107060 con 0x7f96140f51a0 2024-01-21T20:02:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.866+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96040095f0 con 0x7f96140f51a0 2024-01-21T20:02:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.866+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614045ad0 con 0x7f96140f51a0 2024-01-21T20:02:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.866+0000 7f9602ffd700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f95fc02ff60 0x7f95fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.866+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9604041100 con 0x7f96140f51a0 2024-01-21T20:02:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.867+0000 7f9611d9b700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f95fc02ff60 0x7f95fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:44.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.867+0000 7f9611d9b700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f95fc02ff60 0x7f95fc032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:44.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:44.871+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9604014420 con 0x7f96140f51a0 2024-01-21T20:02:45.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.077+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9614107f10 con 0x7f96140f51a0 2024-01-21T20:02:45.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.077+0000 7f9602ffd700 1 -- 172.21.15.153:0/622629782 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9604010230 con 0x7f96140f51a0 2024-01-21T20:02:45.068 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:45.068 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f95fc02ff60 msgr2=0x7f95fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f95fc02ff60 0x7f95fc032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).stop 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 msgr2=0x7f96141064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96141064e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f9604013510 tx=0x7f960400b040).stop 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 shutdown_connections 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f95fc02ff60 0x7f95fc032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 --2- 172.21.15.153:0/622629782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96140f51a0 0x7f96141064e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 >> 172.21.15.153:0/622629782 conn(0x7f96140f09a0 msgr2=0x7f961405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 shutdown_connections 2024-01-21T20:02:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:45.079+0000 7f961359e700 1 -- 172.21.15.153:0/622629782 wait complete. 2024-01-21T20:02:45.070 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:45.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:44.674555+0000 mgr.smithi119.rqacbm (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:46.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:45 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:45.077841+0000 mon.smithi119 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.153:0/622629782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:46.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:46.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:46.674891+0000 mgr.smithi119.rqacbm (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.664+0000 7f84ab345700 1 -- 172.21.15.153:0/396440532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 msgr2=0x7f84a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.664+0000 7f84ab345700 1 --2- 172.21.15.153:0/396440532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a40f55c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f8494009fb0 tx=0x7f8494009b30).stop 2024-01-21T20:02:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.664+0000 7f84ab345700 1 -- 172.21.15.153:0/396440532 shutdown_connections 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.664+0000 7f84ab345700 1 --2- 172.21.15.153:0/396440532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a40f55c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.664+0000 7f84ab345700 1 -- 172.21.15.153:0/396440532 >> 172.21.15.153:0/396440532 conn(0x7f84a40f09a0 msgr2=0x7f84a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84ab345700 1 -- 172.21.15.153:0/396440532 shutdown_connections 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84ab345700 1 -- 172.21.15.153:0/396440532 wait complete. 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84ab345700 1 Processor -- start 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84ab345700 1 -- start start 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84ab345700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a41042a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84ab345700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4104790 con 0x7f84a40f51a0 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84aa343700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a41042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84aa343700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a41042a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:48008/0 (socket says 172.21.15.153:48008) 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.665+0000 7f84aa343700 1 -- 172.21.15.153:0/2437402041 learned_addr learned my addr 172.21.15.153:0/2437402041 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f84aa343700 1 -- 172.21.15.153:0/2437402041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494005040 con 0x7f84a40f51a0 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f84aa343700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a41042a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8494013510 tx=0x7f849400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:47.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8494009320 con 0x7f84a40f51a0 2024-01-21T20:02:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4104990 con 0x7f84a40f51a0 2024-01-21T20:02:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8494013f00 con 0x7f84a40f51a0 2024-01-21T20:02:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8494014dc0 con 0x7f84a40f51a0 2024-01-21T20:02:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.666+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4104e20 con 0x7f84a40f51a0 2024-01-21T20:02:47.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.667+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84940095f0 con 0x7f84a40f51a0 2024-01-21T20:02:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.667+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a4045ad0 con 0x7f84a40f51a0 2024-01-21T20:02:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.667+0000 7f849affd700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f84900321b0 0x7f8490034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.667+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8494040fe0 con 0x7f84a40f51a0 2024-01-21T20:02:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.667+0000 7f84a9b42700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f84900321b0 0x7f8490034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:47.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.667+0000 7f84a9b42700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f84900321b0 0x7f8490034670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f84a00096a0 tx=0x7f84a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:47.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.671+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8494014420 con 0x7f84a40f51a0 2024-01-21T20:02:47.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.876+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84a4105c70 con 0x7f84a40f51a0 2024-01-21T20:02:47.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.877+0000 7f849affd700 1 -- 172.21.15.153:0/2437402041 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8494010230 con 0x7f84a40f51a0 2024-01-21T20:02:47.869 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:47.869 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:47.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f84900321b0 msgr2=0x7f8490034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:47.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f84900321b0 0x7f8490034670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f84a00096a0 tx=0x7f84a0006b40).stop 2024-01-21T20:02:47.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 msgr2=0x7f84a41042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:47.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a41042a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8494013510 tx=0x7f849400b040).stop 2024-01-21T20:02:47.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 shutdown_connections 2024-01-21T20:02:47.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f84900321b0 0x7f8490034670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:47.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 --2- 172.21.15.153:0/2437402041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84a40f51a0 0x7f84a41042a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:47.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 >> 172.21.15.153:0/2437402041 conn(0x7f84a40f09a0 msgr2=0x7f84a40f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:47.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 shutdown_connections 2024-01-21T20:02:47.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:47.879+0000 7f84ab345700 1 -- 172.21.15.153:0/2437402041 wait complete. 2024-01-21T20:02:47.871 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:48.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:47.878456+0000 mon.smithi119 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.153:0/2437402041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:49.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:49.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:49.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:48.675264+0000 mgr.smithi119.rqacbm (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:50.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 -- 172.21.15.153:0/3563258013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 msgr2=0x7efe040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 --2- 172.21.15.153:0/3563258013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe040f55c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7efdf4009fb0 tx=0x7efdf4009b30).stop 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 -- 172.21.15.153:0/3563258013 shutdown_connections 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 --2- 172.21.15.153:0/3563258013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe040f55c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 -- 172.21.15.153:0/3563258013 >> 172.21.15.153:0/3563258013 conn(0x7efe040f09a0 msgr2=0x7efe040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 -- 172.21.15.153:0/3563258013 shutdown_connections 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.461+0000 7efe03fff700 1 -- 172.21.15.153:0/3563258013 wait complete. 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe03fff700 1 Processor -- start 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe03fff700 1 -- start start 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe03fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe041053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:50.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe03fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe041058d0 con 0x7efe040f51a0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe02ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe041053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe02ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe041053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:48018/0 (socket says 172.21.15.153:48018) 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.462+0000 7efe02ffd700 1 -- 172.21.15.153:0/903759209 learned_addr learned my addr 172.21.15.153:0/903759209 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efe02ffd700 1 -- 172.21.15.153:0/903759209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4005040 con 0x7efe040f51a0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efe02ffd700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe041053e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7efdf4013510 tx=0x7efdf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf4009320 con 0x7efe040f51a0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdf4013f00 con 0x7efe040f51a0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe04105ad0 con 0x7efe040f51a0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdf4014dc0 con 0x7efe040f51a0 2024-01-21T20:02:50.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.463+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe04105f60 con 0x7efe040f51a0 2024-01-21T20:02:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.464+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdf401d3d0 con 0x7efe040f51a0 2024-01-21T20:02:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.464+0000 7efdeb7fe700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efdec02ff60 0x7efdec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.464+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdf4041110 con 0x7efe040f51a0 2024-01-21T20:02:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.464+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe04045ad0 con 0x7efe040f51a0 2024-01-21T20:02:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.464+0000 7efe027fc700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efdec02ff60 0x7efdec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:50.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.465+0000 7efe027fc700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efdec02ff60 0x7efdec032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7efdf80096a0 tx=0x7efdf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:50.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.469+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdf4012920 con 0x7efe040f51a0 2024-01-21T20:02:50.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.673+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe04106cb0 con 0x7efe040f51a0 2024-01-21T20:02:50.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.673+0000 7efdeb7fe700 1 -- 172.21.15.153:0/903759209 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efdf402b020 con 0x7efe040f51a0 2024-01-21T20:02:50.665 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:50.665 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:50.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efdec02ff60 msgr2=0x7efdec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efdec02ff60 0x7efdec032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7efdf80096a0 tx=0x7efdf8006b40).stop 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 msgr2=0x7efe041053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe041053e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7efdf4013510 tx=0x7efdf400b040).stop 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 shutdown_connections 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efdec02ff60 0x7efdec032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 --2- 172.21.15.153:0/903759209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe040f51a0 0x7efe041053e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 >> 172.21.15.153:0/903759209 conn(0x7efe040f09a0 msgr2=0x7efe040f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 shutdown_connections 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:50.675+0000 7efe03fff700 1 -- 172.21.15.153:0/903759209 wait complete. 2024-01-21T20:02:50.667 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:50.674946+0000 mon.smithi119 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.153:0/903759209' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:50.675547+0000 mgr.smithi119.rqacbm (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:52.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:52.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:53.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:52.675789+0000 mgr.smithi119.rqacbm (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:53.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 -- 172.21.15.153:0/3858950571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 msgr2=0x7faffc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 --2- 172.21.15.153:0/3858950571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc0f66c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fafec009fb0 tx=0x7fafec009b30).stop 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 -- 172.21.15.153:0/3858950571 shutdown_connections 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 --2- 172.21.15.153:0/3858950571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc0f66c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 -- 172.21.15.153:0/3858950571 >> 172.21.15.153:0/3858950571 conn(0x7faffc0f1aa0 msgr2=0x7faffc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 -- 172.21.15.153:0/3858950571 shutdown_connections 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 -- 172.21.15.153:0/3858950571 wait complete. 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 Processor -- start 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.313+0000 7faffb59e700 1 -- start start 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1069a0 con 0x7faffc0f62a0 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffa59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffa59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56512/0 (socket says 172.21.15.153:56512) 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffa59c700 1 -- 172.21.15.153:0/2579305280 learned_addr learned my addr 172.21.15.153:0/2579305280 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:53.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffa59c700 1 -- 172.21.15.153:0/2579305280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec005040 con 0x7faffc0f62a0 2024-01-21T20:02:53.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffa59c700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc1064b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fafec013510 tx=0x7fafec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:53.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec009320 con 0x7faffc0f62a0 2024-01-21T20:02:53.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc106ba0 con 0x7faffc0f62a0 2024-01-21T20:02:53.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafec013f00 con 0x7faffc0f62a0 2024-01-21T20:02:53.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafec014dc0 con 0x7faffc0f62a0 2024-01-21T20:02:53.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.314+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc107030 con 0x7faffc0f62a0 2024-01-21T20:02:53.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.315+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafec0095f0 con 0x7faffc0f62a0 2024-01-21T20:02:53.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.315+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc045ad0 con 0x7faffc0f62a0 2024-01-21T20:02:53.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.315+0000 7fafeaffd700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fafe402ff60 0x7fafe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:53.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.315+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafec041000 con 0x7faffc0f62a0 2024-01-21T20:02:53.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.316+0000 7faff9d9b700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fafe402ff60 0x7fafe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:53.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.316+0000 7faff9d9b700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fafe402ff60 0x7fafe4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7faff00096a0 tx=0x7faff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:53.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.320+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafec014420 con 0x7faffc0f62a0 2024-01-21T20:02:53.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.526+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faffc107ee0 con 0x7faffc0f62a0 2024-01-21T20:02:53.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.526+0000 7fafeaffd700 1 -- 172.21.15.153:0/2579305280 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fafec010230 con 0x7faffc0f62a0 2024-01-21T20:02:53.518 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:53.518 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fafe402ff60 msgr2=0x7fafe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fafe402ff60 0x7fafe4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7faff00096a0 tx=0x7faff0006b40).stop 2024-01-21T20:02:53.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 msgr2=0x7faffc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc1064b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fafec013510 tx=0x7fafec00b040).stop 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 shutdown_connections 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fafe402ff60 0x7fafe4032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 --2- 172.21.15.153:0/2579305280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faffc0f62a0 0x7faffc1064b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 >> 172.21.15.153:0/2579305280 conn(0x7faffc0f1aa0 msgr2=0x7faffc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 shutdown_connections 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:53.528+0000 7faffb59e700 1 -- 172.21.15.153:0/2579305280 wait complete. 2024-01-21T20:02:53.520 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:54.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:53.527833+0000 mon.smithi119 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.153:0/2579305280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:54.911 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:54.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:55.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:54.676021+0000 mgr.smithi119.rqacbm (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 -- 172.21.15.153:0/1654667322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 msgr2=0x7ff5740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 --2- 172.21.15.153:0/1654667322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7ff568009fb0 tx=0x7ff568009b30).stop 2024-01-21T20:02:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 -- 172.21.15.153:0/1654667322 shutdown_connections 2024-01-21T20:02:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 --2- 172.21.15.153:0/1654667322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 -- 172.21.15.153:0/1654667322 >> 172.21.15.153:0/1654667322 conn(0x7ff5740f0b70 msgr2=0x7ff5740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:56.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 -- 172.21.15.153:0/1654667322 shutdown_connections 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 -- 172.21.15.153:0/1654667322 wait complete. 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 Processor -- start 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.146+0000 7ff57b3c4700 1 -- start start 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff57b3c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff57b3c4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5740fb820 con 0x7ff5740f5410 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff579160700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff579160700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56534/0 (socket says 172.21.15.153:56534) 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff579160700 1 -- 172.21.15.153:0/3599538091 learned_addr learned my addr 172.21.15.153:0/3599538091 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff579160700 1 -- 172.21.15.153:0/3599538091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568005040 con 0x7ff5740f5410 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff579160700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740fd3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff568014000 tx=0x7ff56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5680095f0 con 0x7ff5740f5410 2024-01-21T20:02:56.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5740fb9c0 con 0x7ff5740f5410 2024-01-21T20:02:56.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff568007e70 con 0x7ff5740f5410 2024-01-21T20:02:56.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.147+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff568014ea0 con 0x7ff5740f5410 2024-01-21T20:02:56.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.148+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5740fbe50 con 0x7ff5740f5410 2024-01-21T20:02:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.148+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff568025070 con 0x7ff5740f5410 2024-01-21T20:02:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.148+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff574059100 con 0x7ff5740f5410 2024-01-21T20:02:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.149+0000 7ff565ffb700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff56002ff60 0x7ff560032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.149+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff568040a30 con 0x7ff5740f5410 2024-01-21T20:02:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.149+0000 7ff57895f700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff56002ff60 0x7ff560032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:56.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.149+0000 7ff57895f700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff56002ff60 0x7ff560032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff5700096a0 tx=0x7ff570006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:56.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.153+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5680079c0 con 0x7ff5740f5410 2024-01-21T20:02:56.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.358+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5740fae60 con 0x7ff5740f5410 2024-01-21T20:02:56.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.359+0000 7ff565ffb700 1 -- 172.21.15.153:0/3599538091 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff56800f300 con 0x7ff5740f5410 2024-01-21T20:02:56.350 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:56.350 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:56.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff56002ff60 msgr2=0x7ff560032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff56002ff60 0x7ff560032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff5700096a0 tx=0x7ff570006b40).stop 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 msgr2=0x7ff5740fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740fd3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff568014000 tx=0x7ff56800b040).stop 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 shutdown_connections 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff56002ff60 0x7ff560032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 --2- 172.21.15.153:0/3599538091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5740f5410 0x7ff5740fd3a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 >> 172.21.15.153:0/3599538091 conn(0x7ff5740f0b70 msgr2=0x7ff57405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 shutdown_connections 2024-01-21T20:02:56.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:56.360+0000 7ff57b3c4700 1 -- 172.21.15.153:0/3599538091 wait complete. 2024-01-21T20:02:56.353 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:02:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:56.360257+0000 mon.smithi119 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.153:0/3599538091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:02:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:56.676306+0000 mgr.smithi119.rqacbm (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:57.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:02:57.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:02:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 -- 172.21.15.153:0/2702431199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 msgr2=0x7efc800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 --2- 172.21.15.153:0/2702431199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc800f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7efc70009fb0 tx=0x7efc70009b30).stop 2024-01-21T20:02:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 -- 172.21.15.153:0/2702431199 shutdown_connections 2024-01-21T20:02:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 --2- 172.21.15.153:0/2702431199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc800f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 -- 172.21.15.153:0/2702431199 >> 172.21.15.153:0/2702431199 conn(0x7efc800f0b70 msgr2=0x7efc800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 -- 172.21.15.153:0/2702431199 shutdown_connections 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 -- 172.21.15.153:0/2702431199 wait complete. 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.957+0000 7efc86700700 1 Processor -- start 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc86700700 1 -- start start 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc86700700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc80106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc86700700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc80106c20 con 0x7efc800f5410 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc7ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc80106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc7ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc80106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56550/0 (socket says 172.21.15.153:56550) 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc7ffff700 1 -- 172.21.15.153:0/1113784702 learned_addr learned my addr 172.21.15.153:0/1113784702 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc7ffff700 1 -- 172.21.15.153:0/1113784702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc70005040 con 0x7efc800f5410 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.958+0000 7efc7ffff700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc80106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7efc70013510 tx=0x7efc7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc70009320 con 0x7efc800f5410 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc80106e20 con 0x7efc800f5410 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc70013f00 con 0x7efc800f5410 2024-01-21T20:02:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc70014d90 con 0x7efc800f5410 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc801072b0 con 0x7efc800f5410 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc70014420 con 0x7efc800f5410 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.959+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc80045ad0 con 0x7efc800f5410 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.960+0000 7efc7cff9700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc6802ff60 0x7efc68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.960+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc700410c0 con 0x7efc800f5410 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.960+0000 7efc7f7fe700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc6802ff60 0x7efc68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:02:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.960+0000 7efc7f7fe700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc6802ff60 0x7efc68032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7efc740096a0 tx=0x7efc74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:02:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:58.964+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc700095f0 con 0x7efc800f5410 2024-01-21T20:02:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:02:58.676641+0000 mgr.smithi119.rqacbm (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:02:59.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.168+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc800fae60 con 0x7efc800f5410 2024-01-21T20:02:59.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.169+0000 7efc7cff9700 1 -- 172.21.15.153:0/1113784702 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efc7001db70 con 0x7efc800f5410 2024-01-21T20:02:59.160 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:02:59.160 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc6802ff60 msgr2=0x7efc68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc6802ff60 0x7efc68032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7efc740096a0 tx=0x7efc74006b40).stop 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 msgr2=0x7efc80106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc80106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7efc70013510 tx=0x7efc7000b040).stop 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 shutdown_connections 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efc6802ff60 0x7efc68032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 --2- 172.21.15.153:0/1113784702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc800f5410 0x7efc80106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 >> 172.21.15.153:0/1113784702 conn(0x7efc800f0b70 msgr2=0x7efc800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 shutdown_connections 2024-01-21T20:02:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:02:59.170+0000 7efc86700700 1 -- 172.21.15.153:0/1113784702 wait complete. 2024-01-21T20:02:59.163 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:00.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:58.782329+0000 mon.smithi119 (mon.0) 173 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:03:00.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:58.782980+0000 mon.smithi119 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:03:00.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:02:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:02:59.170023+0000 mon.smithi119 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.153:0/1113784702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:00.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:00.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:01.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:00.676917+0000 mgr.smithi119.rqacbm (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:01.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/1768738321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 msgr2=0x7fb8c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:01.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 --2- 172.21.15.153:0/1768738321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c80f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb8bc009fb0 tx=0x7fb8bc009b30).stop 2024-01-21T20:03:01.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/1768738321 shutdown_connections 2024-01-21T20:03:01.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 --2- 172.21.15.153:0/1768738321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c80f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:01.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/1768738321 >> 172.21.15.153:0/1768738321 conn(0x7fb8c80f0b70 msgr2=0x7fb8c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:01.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/1768738321 shutdown_connections 2024-01-21T20:03:01.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.765+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/1768738321 wait complete. 2024-01-21T20:03:01.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8ce3ce700 1 Processor -- start 2024-01-21T20:03:01.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8ce3ce700 1 -- start start 2024-01-21T20:03:01.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8ce3ce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c810b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:01.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8ce3ce700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c81071e0 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8c7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c810b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8c7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c810b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56566/0 (socket says 172.21.15.153:56566) 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.766+0000 7fb8c7fff700 1 -- 172.21.15.153:0/3545691227 learned_addr learned my addr 172.21.15.153:0/3545691227 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8c7fff700 1 -- 172.21.15.153:0/3545691227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc005040 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8c7fff700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c810b7f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb8bc013510 tx=0x7fb8bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc009320 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8060220 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8bc013f00 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8bc014d90 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.767+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c810bce0 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.768+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8bc0263a0 con 0x7fb8c80f5410 2024-01-21T20:03:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.768+0000 7fb8c4ff9700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:01.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.768+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8bc0410d0 con 0x7fb8c80f5410 2024-01-21T20:03:01.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.768+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c8045ad0 con 0x7fb8c80f5410 2024-01-21T20:03:01.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.768+0000 7fb8c77fe700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:01.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.768+0000 7fb8c77fe700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8b002ff60 0x7fb8b0032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb8b80096a0 tx=0x7fb8b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:01.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.773+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8bc012920 con 0x7fb8c80f5410 2024-01-21T20:03:01.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.978+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c810c390 con 0x7fb8c80f5410 2024-01-21T20:03:01.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.979+0000 7fb8c4ff9700 1 -- 172.21.15.153:0/3545691227 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb8bc02b020 con 0x7fb8c80f5410 2024-01-21T20:03:01.970 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:01.970 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8b002ff60 msgr2=0x7fb8b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8b002ff60 0x7fb8b0032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb8b80096a0 tx=0x7fb8b8006b40).stop 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 msgr2=0x7fb8c810b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c810b7f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb8bc013510 tx=0x7fb8bc00b040).stop 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 shutdown_connections 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8b002ff60 0x7fb8b0032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 --2- 172.21.15.153:0/3545691227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8c80f5410 0x7fb8c810b7f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.980+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 >> 172.21.15.153:0/3545691227 conn(0x7fb8c80f0b70 msgr2=0x7fb8c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.981+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 shutdown_connections 2024-01-21T20:03:01.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:01.981+0000 7fb8ce3ce700 1 -- 172.21.15.153:0/3545691227 wait complete. 2024-01-21T20:03:01.973 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:02.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:01.980173+0000 mon.smithi119 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.153:0/3545691227' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:03.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:03.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:03.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:02.677250+0000 mgr.smithi119.rqacbm (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- 172.21.15.153:0/703885289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 msgr2=0x7f90200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:04.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 --2- 172.21.15.153:0/703885289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90200f66c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f9014009fb0 tx=0x7f9014009b30).stop 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- 172.21.15.153:0/703885289 shutdown_connections 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 --2- 172.21.15.153:0/703885289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90200f66c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- 172.21.15.153:0/703885289 >> 172.21.15.153:0/703885289 conn(0x7f90200f1aa0 msgr2=0x7f90200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- 172.21.15.153:0/703885289 shutdown_connections 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- 172.21.15.153:0/703885289 wait complete. 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 Processor -- start 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- start start 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90201064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.618+0000 7f901f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90201069a0 con 0x7f90200f62a0 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90201064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90201064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38740/0 (socket says 172.21.15.153:38740) 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901e59c700 1 -- 172.21.15.153:0/58273819 learned_addr learned my addr 172.21.15.153:0/58273819 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901e59c700 1 -- 172.21.15.153:0/58273819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9014005040 con 0x7f90200f62a0 2024-01-21T20:03:04.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901e59c700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90201064b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9014013510 tx=0x7f901400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9014009320 con 0x7f90200f62a0 2024-01-21T20:03:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9014013f00 con 0x7f90200f62a0 2024-01-21T20:03:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9020106ba0 con 0x7f90200f62a0 2024-01-21T20:03:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9014014dc0 con 0x7f90200f62a0 2024-01-21T20:03:04.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.619+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9020107030 con 0x7f90200f62a0 2024-01-21T20:03:04.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.620+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90140095f0 con 0x7f90200f62a0 2024-01-21T20:03:04.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.620+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9020045ad0 con 0x7f90200f62a0 2024-01-21T20:03:04.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.620+0000 7f900effd700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f900802ff60 0x7f9008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:04.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.620+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9014041000 con 0x7f90200f62a0 2024-01-21T20:03:04.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.621+0000 7f901dd9b700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f900802ff60 0x7f9008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:04.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.621+0000 7f901dd9b700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f900802ff60 0x7f9008032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f90100096a0 tx=0x7f9010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:04.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.625+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f901401c790 con 0x7f90200f62a0 2024-01-21T20:03:04.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.831+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9020107ee0 con 0x7f90200f62a0 2024-01-21T20:03:04.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.832+0000 7f900effd700 1 -- 172.21.15.153:0/58273819 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9014010230 con 0x7f90200f62a0 2024-01-21T20:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f900802ff60 msgr2=0x7f9008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f900802ff60 0x7f9008032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f90100096a0 tx=0x7f9010006b40).stop 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 msgr2=0x7f90201064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90201064b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9014013510 tx=0x7f901400b040).stop 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 shutdown_connections 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f900802ff60 0x7f9008032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 --2- 172.21.15.153:0/58273819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90200f62a0 0x7f90201064b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 >> 172.21.15.153:0/58273819 conn(0x7f90200f1aa0 msgr2=0x7f90200f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 shutdown_connections 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:04.833+0000 7f901f59e700 1 -- 172.21.15.153:0/58273819 wait complete. 2024-01-21T20:03:04.825 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:05.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:04.677546+0000 mgr.smithi119.rqacbm (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:06.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:04.833075+0000 mon.smithi119 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.153:0/58273819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:06.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:06.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 -- 172.21.15.153:0/985375969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 msgr2=0x7fc2240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 --2- 172.21.15.153:0/985375969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc2240f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fc214009fb0 tx=0x7fc214009b30).stop 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 -- 172.21.15.153:0/985375969 shutdown_connections 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 --2- 172.21.15.153:0/985375969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc2240f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 -- 172.21.15.153:0/985375969 >> 172.21.15.153:0/985375969 conn(0x7fc2240f0b70 msgr2=0x7fc2240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 -- 172.21.15.153:0/985375969 shutdown_connections 2024-01-21T20:03:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.433+0000 7fc22a5cd700 1 -- 172.21.15.153:0/985375969 wait complete. 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.434+0000 7fc22a5cd700 1 Processor -- start 2024-01-21T20:03:07.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.434+0000 7fc22a5cd700 1 -- start start 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc22a5cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc224105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc22a5cd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc224105b20 con 0x7fc2240f5410 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc223fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc224105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc223fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc224105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38756/0 (socket says 172.21.15.153:38756) 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc223fff700 1 -- 172.21.15.153:0/3782709692 learned_addr learned my addr 172.21.15.153:0/3782709692 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc223fff700 1 -- 172.21.15.153:0/3782709692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214005040 con 0x7fc2240f5410 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc223fff700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc224105630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc214013510 tx=0x7fc21400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc214009320 con 0x7fc2240f5410 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc214013f00 con 0x7fc2240f5410 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.435+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc224105d20 con 0x7fc2240f5410 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.436+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc214014d90 con 0x7fc2240f5410 2024-01-21T20:03:07.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.436+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2241061b0 con 0x7fc2240f5410 2024-01-21T20:03:07.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.436+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc214014420 con 0x7fc2240f5410 2024-01-21T20:03:07.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.436+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc224045ad0 con 0x7fc2240f5410 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.436+0000 7fc220ff9700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc20c02ff60 0x7fc20c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.436+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2140410c0 con 0x7fc2240f5410 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.437+0000 7fc2237fe700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc20c02ff60 0x7fc20c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:07.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.437+0000 7fc2237fe700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc20c02ff60 0x7fc20c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc2180096a0 tx=0x7fc218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:07.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.441+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2140095f0 con 0x7fc2240f5410 2024-01-21T20:03:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:06.677858+0000 mgr.smithi119.rqacbm (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:07.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.644+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2240fae60 con 0x7fc2240f5410 2024-01-21T20:03:07.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.645+0000 7fc220ff9700 1 -- 172.21.15.153:0/3782709692 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc21401db70 con 0x7fc2240f5410 2024-01-21T20:03:07.636 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:07.636 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.646+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc20c02ff60 msgr2=0x7fc20c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.646+0000 7fc22a5cd700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc20c02ff60 0x7fc20c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc2180096a0 tx=0x7fc218006b40).stop 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.646+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 msgr2=0x7fc224105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.646+0000 7fc22a5cd700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc224105630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc214013510 tx=0x7fc21400b040).stop 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.646+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 shutdown_connections 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.647+0000 7fc22a5cd700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc20c02ff60 0x7fc20c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.647+0000 7fc22a5cd700 1 --2- 172.21.15.153:0/3782709692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2240f5410 0x7fc224105630 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.647+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 >> 172.21.15.153:0/3782709692 conn(0x7fc2240f0b70 msgr2=0x7fc2240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:07.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.647+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 shutdown_connections 2024-01-21T20:03:07.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:07.647+0000 7fc22a5cd700 1 -- 172.21.15.153:0/3782709692 wait complete. 2024-01-21T20:03:07.639 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:08.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:07.646561+0000 mon.smithi119 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.153:0/3782709692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:08.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:08.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:09.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:08.678171+0000 mgr.smithi119.rqacbm (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 -- 172.21.15.153:0/2861755252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 msgr2=0x7f38b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 --2- 172.21.15.153:0/2861755252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b00f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f38a0009fb0 tx=0x7f38a0009b30).stop 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 -- 172.21.15.153:0/2861755252 shutdown_connections 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 --2- 172.21.15.153:0/2861755252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b00f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 -- 172.21.15.153:0/2861755252 >> 172.21.15.153:0/2861755252 conn(0x7f38b00f0b70 msgr2=0x7f38b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 -- 172.21.15.153:0/2861755252 shutdown_connections 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 -- 172.21.15.153:0/2861755252 wait complete. 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 Processor -- start 2024-01-21T20:03:10.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.140+0000 7f38b5c15700 1 -- start start 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38b5c15700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38b5c15700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b0106c10 con 0x7f38b00f5410 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38786/0 (socket says 172.21.15.153:38786) 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38af7fe700 1 -- 172.21.15.153:0/1925667219 learned_addr learned my addr 172.21.15.153:0/1925667219 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38af7fe700 1 -- 172.21.15.153:0/1925667219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a0005040 con 0x7f38b00f5410 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38af7fe700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f38a0013510 tx=0x7f38a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a0009320 con 0x7f38b00f5410 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b0106e10 con 0x7f38b00f5410 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38a0013f00 con 0x7f38b00f5410 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a0014d90 con 0x7f38b00f5410 2024-01-21T20:03:10.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.141+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01072a0 con 0x7f38b00f5410 2024-01-21T20:03:10.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.142+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38a00095f0 con 0x7f38b00f5410 2024-01-21T20:03:10.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.142+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b0045ad0 con 0x7f38b00f5410 2024-01-21T20:03:10.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.142+0000 7f3897fff700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f389802ff60 0x7f3898032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:10.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.142+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38a0041000 con 0x7f38b00f5410 2024-01-21T20:03:10.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.143+0000 7f38aeffd700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f389802ff60 0x7f3898032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:10.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.143+0000 7f38aeffd700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f389802ff60 0x7f3898032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f38a40096a0 tx=0x7f38a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:10.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.147+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38a0014420 con 0x7f38b00f5410 2024-01-21T20:03:10.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.357+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38b0108150 con 0x7f38b00f5410 2024-01-21T20:03:10.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.357+0000 7f3897fff700 1 -- 172.21.15.153:0/1925667219 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f38a001d7f0 con 0x7f38b00f5410 2024-01-21T20:03:10.349 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:10.349 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:10.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f389802ff60 msgr2=0x7f3898032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f389802ff60 0x7f3898032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f38a40096a0 tx=0x7f38a4006b40).stop 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 msgr2=0x7f38b0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f38a0013510 tx=0x7f38a000b040).stop 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 shutdown_connections 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f389802ff60 0x7f3898032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 --2- 172.21.15.153:0/1925667219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38b00f5410 0x7f38b0106720 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 >> 172.21.15.153:0/1925667219 conn(0x7f38b00f0b70 msgr2=0x7f38b005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 shutdown_connections 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:10.359+0000 7f38b5c15700 1 -- 172.21.15.153:0/1925667219 wait complete. 2024-01-21T20:03:10.351 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:10.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:10.358846+0000 mon.smithi119 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.153:0/1925667219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:11.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:11.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:11.838 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:10.678447+0000 mgr.smithi119.rqacbm (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 -- 172.21.15.153:0/3755079357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 msgr2=0x7fd9dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 --2- 172.21.15.153:0/3755079357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0f55c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fd9d8009fb0 tx=0x7fd9d8009b30).stop 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 -- 172.21.15.153:0/3755079357 shutdown_connections 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 --2- 172.21.15.153:0/3755079357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0f55c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 -- 172.21.15.153:0/3755079357 >> 172.21.15.153:0/3755079357 conn(0x7fd9dc0f09a0 msgr2=0x7fd9dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 -- 172.21.15.153:0/3755079357 shutdown_connections 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 -- 172.21.15.153:0/3755079357 wait complete. 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.948+0000 7fd9e2302700 1 Processor -- start 2024-01-21T20:03:12.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e2302700 1 -- start start 2024-01-21T20:03:12.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e2302700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:12.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e2302700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc0fb5a0 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e1300700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:12.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e1300700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38798/0 (socket says 172.21.15.153:38798) 2024-01-21T20:03:12.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e1300700 1 -- 172.21.15.153:0/3941553105 learned_addr learned my addr 172.21.15.153:0/3941553105 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e1300700 1 -- 172.21.15.153:0/3941553105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8005040 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9e1300700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0fd120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd9d8013510 tx=0x7fd9d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.949+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d8009320 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.950+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc0fb740 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.950+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9d8013f00 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.950+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d8014dc0 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.950+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc0fbbd0 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.950+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9d80095f0 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.951+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc059100 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.951+0000 7fd9d1ffb700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9c802ff60 0x7fd9c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:12.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.951+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9d80410c0 con 0x7fd9dc0f51a0 2024-01-21T20:03:12.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.951+0000 7fd9e0aff700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9c802ff60 0x7fd9c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:12.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.951+0000 7fd9e0aff700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9c802ff60 0x7fd9c8032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd9cc0096a0 tx=0x7fd9cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:12.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:12.955+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9d8014420 con 0x7fd9dc0f51a0 2024-01-21T20:03:13.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:12.678780+0000 mgr.smithi119.rqacbm (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:13.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.162+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9dc0fabe0 con 0x7fd9dc0f51a0 2024-01-21T20:03:13.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.163+0000 7fd9d1ffb700 1 -- 172.21.15.153:0/3941553105 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd9d8010230 con 0x7fd9dc0f51a0 2024-01-21T20:03:13.154 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:13.154 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:13.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9c802ff60 msgr2=0x7fd9c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:13.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9c802ff60 0x7fd9c8032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd9cc0096a0 tx=0x7fd9cc006b40).stop 2024-01-21T20:03:13.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 msgr2=0x7fd9dc0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:13.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0fd120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd9d8013510 tx=0x7fd9d800b040).stop 2024-01-21T20:03:13.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 shutdown_connections 2024-01-21T20:03:13.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9c802ff60 0x7fd9c8032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:13.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 --2- 172.21.15.153:0/3941553105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f51a0 0x7fd9dc0fd120 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:13.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 >> 172.21.15.153:0/3941553105 conn(0x7fd9dc0f09a0 msgr2=0x7fd9dc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:13.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 shutdown_connections 2024-01-21T20:03:13.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:13.165+0000 7fd9e2302700 1 -- 172.21.15.153:0/3941553105 wait complete. 2024-01-21T20:03:13.157 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:14.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:13.164712+0000 mon.smithi119 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.153:0/3941553105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:14.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:14.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:14.679220+0000 mgr.smithi119.rqacbm (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:15.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3681935161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 msgr2=0x7fc8e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 --2- 172.21.15.153:0/3681935161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e00f66c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fc8dc009fb0 tx=0x7fc8dc009b30).stop 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3681935161 shutdown_connections 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 --2- 172.21.15.153:0/3681935161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e00f66c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3681935161 >> 172.21.15.153:0/3681935161 conn(0x7fc8e00f1aa0 msgr2=0x7fc8e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3681935161 shutdown_connections 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3681935161 wait complete. 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 Processor -- start 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.682+0000 7fc8e73ab700 1 -- start start 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e73ab700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e73ab700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0069e20 con 0x7fc8e00f62a0 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e63a9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:15.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e63a9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:36866/0 (socket says 172.21.15.153:36866) 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e63a9700 1 -- 172.21.15.153:0/3081066388 learned_addr learned my addr 172.21.15.153:0/3081066388 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e63a9700 1 -- 172.21.15.153:0/3081066388 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc005040 con 0x7fc8e00f62a0 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e63a9700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e006b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fc8dc013510 tx=0x7fc8dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8dc009320 con 0x7fc8e00f62a0 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e0069fc0 con 0x7fc8e00f62a0 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8dc013f00 con 0x7fc8e00f62a0 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8dc014dc0 con 0x7fc8e00f62a0 2024-01-21T20:03:15.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.683+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e006a450 con 0x7fc8e00f62a0 2024-01-21T20:03:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.684+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e0059100 con 0x7fc8e00f62a0 2024-01-21T20:03:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.684+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8dc0095f0 con 0x7fc8e00f62a0 2024-01-21T20:03:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.684+0000 7fc8d6ffd700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc8cc02ff60 0x7fc8cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.684+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8dc0410e0 con 0x7fc8e00f62a0 2024-01-21T20:03:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.685+0000 7fc8e5ba8700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc8cc02ff60 0x7fc8cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:15.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.685+0000 7fc8e5ba8700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc8cc02ff60 0x7fc8cc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc8d00096a0 tx=0x7fc8d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:15.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.689+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8dc014420 con 0x7fc8e00f62a0 2024-01-21T20:03:15.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.894+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8e00fbce0 con 0x7fc8e00f62a0 2024-01-21T20:03:15.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.894+0000 7fc8d6ffd700 1 -- 172.21.15.153:0/3081066388 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc8dc01c920 con 0x7fc8e00f62a0 2024-01-21T20:03:15.886 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:15.886 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc8cc02ff60 msgr2=0x7fc8cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc8cc02ff60 0x7fc8cc032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fc8d00096a0 tx=0x7fc8d0006b40).stop 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 msgr2=0x7fc8e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e006b9a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fc8dc013510 tx=0x7fc8dc00b040).stop 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 shutdown_connections 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc8cc02ff60 0x7fc8cc032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:15.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 --2- 172.21.15.153:0/3081066388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00f62a0 0x7fc8e006b9a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:15.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 >> 172.21.15.153:0/3081066388 conn(0x7fc8e00f1aa0 msgr2=0x7fc8e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:15.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 shutdown_connections 2024-01-21T20:03:15.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:15.896+0000 7fc8e73ab700 1 -- 172.21.15.153:0/3081066388 wait complete. 2024-01-21T20:03:15.888 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:16.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:15 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:15.895863+0000 mon.smithi119 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.153:0/3081066388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:17.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:17.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:16.679563+0000 mgr.smithi119.rqacbm (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:18.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 -- 172.21.15.153:0/3949412044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 msgr2=0x7f53080f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:18.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 --2- 172.21.15.153:0/3949412044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f8f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f5304009fb0 tx=0x7f5304009b30).stop 2024-01-21T20:03:18.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 -- 172.21.15.153:0/3949412044 shutdown_connections 2024-01-21T20:03:18.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 --2- 172.21.15.153:0/3949412044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f8f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:18.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 -- 172.21.15.153:0/3949412044 >> 172.21.15.153:0/3949412044 conn(0x7f53080f09a0 msgr2=0x7f53080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:18.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 -- 172.21.15.153:0/3949412044 shutdown_connections 2024-01-21T20:03:18.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.490+0000 7f530e33a700 1 -- 172.21.15.153:0/3949412044 wait complete. 2024-01-21T20:03:18.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530e33a700 1 Processor -- start 2024-01-21T20:03:18.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530e33a700 1 -- start start 2024-01-21T20:03:18.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530e33a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:18.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530e33a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53080f4e20 con 0x7f53080601e0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530d338700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530d338700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:36892/0 (socket says 172.21.15.153:36892) 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.491+0000 7f530d338700 1 -- 172.21.15.153:0/3695279864 learned_addr learned my addr 172.21.15.153:0/3695279864 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f530d338700 1 -- 172.21.15.153:0/3695279864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304005040 con 0x7f53080601e0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f530d338700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f69a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5304013510 tx=0x7f530400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5304009320 con 0x7f53080601e0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53080f4fc0 con 0x7f53080601e0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5304013f00 con 0x7f53080601e0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5304014d90 con 0x7f53080601e0 2024-01-21T20:03:18.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.492+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53080f5450 con 0x7f53080601e0 2024-01-21T20:03:18.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.493+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53040095f0 con 0x7f53080601e0 2024-01-21T20:03:18.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.493+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5308059100 con 0x7f53080601e0 2024-01-21T20:03:18.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.493+0000 7f52fdffb700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f52f402ffb0 0x7f52f4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:18.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.493+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5304041050 con 0x7f53080601e0 2024-01-21T20:03:18.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.493+0000 7f530cb37700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f52f402ffb0 0x7f52f4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:18.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.494+0000 7f530cb37700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f52f402ffb0 0x7f52f4032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f52f80096a0 tx=0x7f52f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:18.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.498+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5304014420 con 0x7f53080601e0 2024-01-21T20:03:18.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.702+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53080fbcd0 con 0x7f53080601e0 2024-01-21T20:03:18.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.703+0000 7f52fdffb700 1 -- 172.21.15.153:0/3695279864 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f530401d7f0 con 0x7f53080601e0 2024-01-21T20:03:18.694 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:18.694 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.704+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f52f402ffb0 msgr2=0x7f52f4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.704+0000 7f530e33a700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f52f402ffb0 0x7f52f4032470 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f52f80096a0 tx=0x7f52f8006b40).stop 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 msgr2=0x7f53080f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f69a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f5304013510 tx=0x7f530400b040).stop 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 shutdown_connections 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f52f402ffb0 0x7f52f4032470 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 --2- 172.21.15.153:0/3695279864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53080601e0 0x7f53080f69a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 >> 172.21.15.153:0/3695279864 conn(0x7f53080f09a0 msgr2=0x7f53080f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 shutdown_connections 2024-01-21T20:03:18.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:18.705+0000 7f530e33a700 1 -- 172.21.15.153:0/3695279864 wait complete. 2024-01-21T20:03:18.697 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:19.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:18.679959+0000 mgr.smithi119.rqacbm (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:19.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:18.704422+0000 mon.smithi119 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.153:0/3695279864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:20.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:20.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:21.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:20.680316+0000 mgr.smithi119.rqacbm (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:21.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 -- 172.21.15.153:0/3724103850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 msgr2=0x7f40280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:21.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 --2- 172.21.15.153:0/3724103850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40280f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f401c009fb0 tx=0x7f401c009b30).stop 2024-01-21T20:03:21.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 -- 172.21.15.153:0/3724103850 shutdown_connections 2024-01-21T20:03:21.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 --2- 172.21.15.153:0/3724103850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40280f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:21.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 -- 172.21.15.153:0/3724103850 >> 172.21.15.153:0/3724103850 conn(0x7f40280f0b70 msgr2=0x7f40280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:21.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 -- 172.21.15.153:0/3724103850 shutdown_connections 2024-01-21T20:03:21.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.274+0000 7f402d564700 1 -- 172.21.15.153:0/3724103850 wait complete. 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f402d564700 1 Processor -- start 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f402d564700 1 -- start start 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f402d564700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40281066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f402d564700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4028106bc0 con 0x7f40280f5410 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f4026ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40281066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f4026ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40281066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:36916/0 (socket says 172.21.15.153:36916) 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f4026ffd700 1 -- 172.21.15.153:0/2432956951 learned_addr learned my addr 172.21.15.153:0/2432956951 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:21.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f4026ffd700 1 -- 172.21.15.153:0/2432956951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c005040 con 0x7f40280f5410 2024-01-21T20:03:21.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.275+0000 7f4026ffd700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40281066d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f401c013510 tx=0x7f401c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:21.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.276+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401c009320 con 0x7f40280f5410 2024-01-21T20:03:21.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.276+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f401c013f00 con 0x7f40280f5410 2024-01-21T20:03:21.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.276+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4028106dc0 con 0x7f40280f5410 2024-01-21T20:03:21.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.276+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f401c014d90 con 0x7f40280f5410 2024-01-21T20:03:21.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.276+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4028107250 con 0x7f40280f5410 2024-01-21T20:03:21.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.276+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f401c0095f0 con 0x7f40280f5410 2024-01-21T20:03:21.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.277+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4028045ad0 con 0x7f40280f5410 2024-01-21T20:03:21.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.277+0000 7f400f7fe700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f401002ff60 0x7f4010032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:21.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.277+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f401c0410c0 con 0x7f40280f5410 2024-01-21T20:03:21.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.277+0000 7f40267fc700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f401002ff60 0x7f4010032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:21.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.277+0000 7f40267fc700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f401002ff60 0x7f4010032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f40180096a0 tx=0x7f4018006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:21.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.281+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f401c01d7f0 con 0x7f40280f5410 2024-01-21T20:03:21.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.488+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4028108100 con 0x7f40280f5410 2024-01-21T20:03:21.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.489+0000 7f400f7fe700 1 -- 172.21.15.153:0/2432956951 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f401c010230 con 0x7f40280f5410 2024-01-21T20:03:21.481 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:21.481 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f401002ff60 msgr2=0x7f4010032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f401002ff60 0x7f4010032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f40180096a0 tx=0x7f4018006b40).stop 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 msgr2=0x7f40281066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40281066d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f401c013510 tx=0x7f401c00b040).stop 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 shutdown_connections 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f401002ff60 0x7f4010032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:21.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 --2- 172.21.15.153:0/2432956951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40280f5410 0x7f40281066d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 >> 172.21.15.153:0/2432956951 conn(0x7f40280f0b70 msgr2=0x7f402805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 shutdown_connections 2024-01-21T20:03:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:21.491+0000 7f402d564700 1 -- 172.21.15.153:0/2432956951 wait complete. 2024-01-21T20:03:21.483 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:22.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:21.490600+0000 mon.smithi119 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.153:0/2432956951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:22.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:22.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:23.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:22.680546+0000 mgr.smithi119.rqacbm (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 -- 172.21.15.153:0/1886891034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 msgr2=0x7ff5140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 --2- 172.21.15.153:0/1886891034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5140f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff4fc009fb0 tx=0x7ff4fc009b30).stop 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 -- 172.21.15.153:0/1886891034 shutdown_connections 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 --2- 172.21.15.153:0/1886891034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5140f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 -- 172.21.15.153:0/1886891034 >> 172.21.15.153:0/1886891034 conn(0x7ff5140f0b70 msgr2=0x7ff5140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 -- 172.21.15.153:0/1886891034 shutdown_connections 2024-01-21T20:03:24.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.094+0000 7ff518fd8700 1 -- 172.21.15.153:0/1886891034 wait complete. 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff518fd8700 1 Processor -- start 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff518fd8700 1 -- start start 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff518fd8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5141093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff518fd8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff514060280 con 0x7ff5140f5410 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff51259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5141093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff51259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5141093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47148/0 (socket says 172.21.15.153:47148) 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.095+0000 7ff51259c700 1 -- 172.21.15.153:0/2281999302 learned_addr learned my addr 172.21.15.153:0/2281999302 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff51259c700 1 -- 172.21.15.153:0/2281999302 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc005040 con 0x7ff5140f5410 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff51259c700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5141093f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff4fc013510 tx=0x7ff4fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4fc009320 con 0x7ff5140f5410 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4fc013f00 con 0x7ff5140f5410 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff514060480 con 0x7ff5140f5410 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4fc014d90 con 0x7ff5140f5410 2024-01-21T20:03:24.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.096+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff514109b20 con 0x7ff5140f5410 2024-01-21T20:03:24.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.097+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4fc014420 con 0x7ff5140f5410 2024-01-21T20:03:24.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.097+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff514045ad0 con 0x7ff5140f5410 2024-01-21T20:03:24.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.097+0000 7ff50affd700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff50002ff60 0x7ff500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.097+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4fc041000 con 0x7ff5140f5410 2024-01-21T20:03:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.097+0000 7ff511d9b700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff50002ff60 0x7ff500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:24.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.098+0000 7ff511d9b700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff50002ff60 0x7ff500032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff5040096a0 tx=0x7ff504006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:24.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.102+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4fc0095f0 con 0x7ff5140f5410 2024-01-21T20:03:24.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.306+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff51410a340 con 0x7ff5140f5410 2024-01-21T20:03:24.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.307+0000 7ff50affd700 1 -- 172.21.15.153:0/2281999302 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff4fc01db70 con 0x7ff5140f5410 2024-01-21T20:03:24.298 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:24.298 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:24.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff50002ff60 msgr2=0x7ff500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff50002ff60 0x7ff500032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff5040096a0 tx=0x7ff504006b40).stop 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 msgr2=0x7ff5141093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5141093f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff4fc013510 tx=0x7ff4fc00b040).stop 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 shutdown_connections 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff50002ff60 0x7ff500032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 --2- 172.21.15.153:0/2281999302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5140f5410 0x7ff5141093f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 >> 172.21.15.153:0/2281999302 conn(0x7ff5140f0b70 msgr2=0x7ff5140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 shutdown_connections 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:24.308+0000 7ff518fd8700 1 -- 172.21.15.153:0/2281999302 wait complete. 2024-01-21T20:03:24.300 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:24.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:24.308089+0000 mon.smithi119 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.153:0/2281999302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:25.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:25.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:25.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:24.680827+0000 mgr.smithi119.rqacbm (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:26.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/1898702594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 msgr2=0x7fb8ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 --2- 172.21.15.153:0/1898702594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fb8dc009fb0 tx=0x7fb8dc009b30).stop 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/1898702594 shutdown_connections 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 --2- 172.21.15.153:0/1898702594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/1898702594 >> 172.21.15.153:0/1898702594 conn(0x7fb8ec0f0b70 msgr2=0x7fb8ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/1898702594 shutdown_connections 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.852+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/1898702594 wait complete. 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8f0e2d700 1 Processor -- start 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8f0e2d700 1 -- start start 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8f0e2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8f0e2d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec105b20 con 0x7fb8ec0f5410 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8ea59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8ea59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47180/0 (socket says 172.21.15.153:47180) 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.853+0000 7fb8ea59c700 1 -- 172.21.15.153:0/3169892287 learned_addr learned my addr 172.21.15.153:0/3169892287 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8ea59c700 1 -- 172.21.15.153:0/3169892287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc005040 con 0x7fb8ec0f5410 2024-01-21T20:03:26.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8ea59c700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb8dc013510 tx=0x7fb8dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:26.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8dc009320 con 0x7fb8ec0f5410 2024-01-21T20:03:26.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8dc013f00 con 0x7fb8ec0f5410 2024-01-21T20:03:26.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec105d20 con 0x7fb8ec0f5410 2024-01-21T20:03:26.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8dc014d90 con 0x7fb8ec0f5410 2024-01-21T20:03:26.847 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.854+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec1061b0 con 0x7fb8ec0f5410 2024-01-21T20:03:26.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.855+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8dc014420 con 0x7fb8ec0f5410 2024-01-21T20:03:26.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.855+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec045ad0 con 0x7fb8ec0f5410 2024-01-21T20:03:26.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.855+0000 7fb8e2ffd700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:26.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.855+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8dc042eb0 con 0x7fb8ec0f5410 2024-01-21T20:03:26.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.855+0000 7fb8e9d9b700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:26.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.856+0000 7fb8e9d9b700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8d802ff60 0x7fb8d8032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb8d40096a0 tx=0x7fb8d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:26.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:26.860+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8dc0095f0 con 0x7fb8ec0f5410 2024-01-21T20:03:27.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.068+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8ec0fae60 con 0x7fb8ec0f5410 2024-01-21T20:03:27.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.068+0000 7fb8e2ffd700 1 -- 172.21.15.153:0/3169892287 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb8dc01fb70 con 0x7fb8ec0f5410 2024-01-21T20:03:27.060 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:27.060 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:27.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8d802ff60 msgr2=0x7fb8d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:27.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8d802ff60 0x7fb8d8032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb8d40096a0 tx=0x7fb8d4006b40).stop 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 msgr2=0x7fb8ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb8dc013510 tx=0x7fb8dc00b040).stop 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 shutdown_connections 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 --2- 172.21.15.153:0/3169892287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8ec0f5410 0x7fb8ec105630 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 >> 172.21.15.153:0/3169892287 conn(0x7fb8ec0f0b70 msgr2=0x7fb8ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 shutdown_connections 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:27.070+0000 7fb8f0e2d700 1 -- 172.21.15.153:0/3169892287 wait complete. 2024-01-21T20:03:27.062 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:26.681120+0000 mgr.smithi119.rqacbm (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:27.069945+0000 mon.smithi119 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.153:0/3169892287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:28.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:28.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:29.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:28.681443+0000 mgr.smithi119.rqacbm (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:29.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 -- 172.21.15.153:0/792774526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 msgr2=0x7f45740f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 --2- 172.21.15.153:0/792774526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f45740f6ea0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f455c009fb0 tx=0x7f455c009b30).stop 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 -- 172.21.15.153:0/792774526 shutdown_connections 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 --2- 172.21.15.153:0/792774526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f45740f6ea0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 -- 172.21.15.153:0/792774526 >> 172.21.15.153:0/792774526 conn(0x7f45740f02e0 msgr2=0x7f45740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 -- 172.21.15.153:0/792774526 shutdown_connections 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 -- 172.21.15.153:0/792774526 wait complete. 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.646+0000 7f4579ef8700 1 Processor -- start 2024-01-21T20:03:29.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f4579ef8700 1 -- start start 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f4579ef8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f4574109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f4579ef8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45740f5310 con 0x7f45740f6a80 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f45737fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f4574109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f45737fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f4574109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47194/0 (socket says 172.21.15.153:47194) 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f45737fe700 1 -- 172.21.15.153:0/1375407815 learned_addr learned my addr 172.21.15.153:0/1375407815 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.647+0000 7f45737fe700 1 -- 172.21.15.153:0/1375407815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455c005040 con 0x7f45740f6a80 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f45737fe700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f4574109430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f455c009830 tx=0x7f455c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455c009320 con 0x7f45740f6a80 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45740f5510 con 0x7f45740f6a80 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f455c0136c0 con 0x7f45740f6a80 2024-01-21T20:03:29.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f455c01ee90 con 0x7f45740f6a80 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4574109ae0 con 0x7f45740f6a80 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.648+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f455c01e420 con 0x7f45740f6a80 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.649+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4574059100 con 0x7f45740f6a80 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.649+0000 7f456bfff700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f456002ff10 0x7f45600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.649+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f455c041080 con 0x7f45740f6a80 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.649+0000 7f4572ffd700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f456002ff10 0x7f45600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:29.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.649+0000 7f4572ffd700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f456002ff10 0x7f45600323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f45640096a0 tx=0x7f4564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:29.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.654+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f455c06d2b0 con 0x7f45740f6a80 2024-01-21T20:03:29.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.859+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4574045ad0 con 0x7f45740f6a80 2024-01-21T20:03:29.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.861+0000 7f456bfff700 1 -- 172.21.15.153:0/1375407815 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f455c02c2d0 con 0x7f45740f6a80 2024-01-21T20:03:29.851 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:29.851 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f456002ff10 msgr2=0x7f45600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f456002ff10 0x7f45600323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f45640096a0 tx=0x7f4564006b40).stop 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 msgr2=0x7f4574109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f4574109430 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f455c009830 tx=0x7f455c00b040).stop 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 shutdown_connections 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f456002ff10 0x7f45600323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 --2- 172.21.15.153:0/1375407815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45740f6a80 0x7f4574109430 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 >> 172.21.15.153:0/1375407815 conn(0x7f45740f02e0 msgr2=0x7f45740f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 shutdown_connections 2024-01-21T20:03:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:29.863+0000 7f4579ef8700 1 -- 172.21.15.153:0/1375407815 wait complete. 2024-01-21T20:03:29.854 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:30.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:29.861255+0000 mon.smithi119 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.153:0/1375407815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:31.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:31.244 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:31.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:30.681844+0000 mgr.smithi119.rqacbm (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:32.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- 172.21.15.153:0/2740018449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 msgr2=0x7f9c040f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:32.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 --2- 172.21.15.153:0/2740018449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c040f66c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f9bf4009fb0 tx=0x7f9bf4009b30).stop 2024-01-21T20:03:32.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- 172.21.15.153:0/2740018449 shutdown_connections 2024-01-21T20:03:32.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 --2- 172.21.15.153:0/2740018449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c040f66c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- 172.21.15.153:0/2740018449 >> 172.21.15.153:0/2740018449 conn(0x7f9c040f1aa0 msgr2=0x7f9c040f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- 172.21.15.153:0/2740018449 shutdown_connections 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- 172.21.15.153:0/2740018449 wait complete. 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 Processor -- start 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- start start 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c041064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.444+0000 7f9c0a542700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c041069a0 con 0x7f9c040f62a0 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c09540700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c041064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c09540700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c041064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47212/0 (socket says 172.21.15.153:47212) 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c09540700 1 -- 172.21.15.153:0/16816706 learned_addr learned my addr 172.21.15.153:0/16816706 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c09540700 1 -- 172.21.15.153:0/16816706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4005040 con 0x7f9c040f62a0 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c09540700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c041064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9bf4013510 tx=0x7f9bf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:32.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf4009320 con 0x7f9c040f62a0 2024-01-21T20:03:32.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bf4013f00 con 0x7f9c040f62a0 2024-01-21T20:03:32.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf4014dc0 con 0x7f9c040f62a0 2024-01-21T20:03:32.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c04106ba0 con 0x7f9c040f62a0 2024-01-21T20:03:32.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.445+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c04107030 con 0x7f9c040f62a0 2024-01-21T20:03:32.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.446+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bf40095f0 con 0x7f9c040f62a0 2024-01-21T20:03:32.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.446+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c04045ad0 con 0x7f9c040f62a0 2024-01-21T20:03:32.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.446+0000 7f9bf9ffb700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bf002ffb0 0x7f9bf0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:32.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.447+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bf4040ff0 con 0x7f9c040f62a0 2024-01-21T20:03:32.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.447+0000 7f9c08d3f700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bf002ffb0 0x7f9bf0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:32.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.447+0000 7f9c08d3f700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bf002ffb0 0x7f9bf0032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9c000096a0 tx=0x7f9c00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:32.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.451+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bf406d2b0 con 0x7f9c040f62a0 2024-01-21T20:03:32.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.656+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c04107ee0 con 0x7f9c040f62a0 2024-01-21T20:03:32.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.657+0000 7f9bf9ffb700 1 -- 172.21.15.153:0/16816706 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9bf4014420 con 0x7f9c040f62a0 2024-01-21T20:03:32.647 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:32.647 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.658+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bf002ffb0 msgr2=0x7f9bf0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bf002ffb0 0x7f9bf0032470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9c000096a0 tx=0x7f9c00006b40).stop 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 msgr2=0x7f9c041064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c041064b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9bf4013510 tx=0x7f9bf400b040).stop 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 shutdown_connections 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bf002ffb0 0x7f9bf0032470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 --2- 172.21.15.153:0/16816706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c040f62a0 0x7f9c041064b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 >> 172.21.15.153:0/16816706 conn(0x7f9c040f1aa0 msgr2=0x7f9c040f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:32.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 shutdown_connections 2024-01-21T20:03:32.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:32.659+0000 7f9c0a542700 1 -- 172.21.15.153:0/16816706 wait complete. 2024-01-21T20:03:32.650 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:33.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:32.657529+0000 mon.smithi119 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.153:0/16816706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:34.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:34.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:34.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:32.682202+0000 mgr.smithi119.rqacbm (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:35.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1217518652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 msgr2=0x7fd5b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:35.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 --2- 172.21.15.153:0/1217518652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b00f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fd5a4009fb0 tx=0x7fd5a4009b30).stop 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1217518652 shutdown_connections 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 --2- 172.21.15.153:0/1217518652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b00f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1217518652 >> 172.21.15.153:0/1217518652 conn(0x7fd5b00f0b70 msgr2=0x7fd5b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1217518652 shutdown_connections 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.217+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1217518652 wait complete. 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5b4fcb700 1 Processor -- start 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5b4fcb700 1 -- start start 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5b4fcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5b4fcb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0060240 con 0x7fd5b00f5410 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56150/0 (socket says 172.21.15.153:56150) 2024-01-21T20:03:35.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.218+0000 7fd5ae59c700 1 -- 172.21.15.153:0/1694988665 learned_addr learned my addr 172.21.15.153:0/1694988665 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd5ae59c700 1 -- 172.21.15.153:0/1694988665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a4005040 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd5ae59c700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b0109430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd5a4014000 tx=0x7fd5a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a40095f0 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b0060440 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5a4007e70 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a4014ea0 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.219+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b0109ae0 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.220+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5a4025070 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.220+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b0045ad0 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.220+0000 7fd59effd700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.220+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5a40409f0 con 0x7fd5b00f5410 2024-01-21T20:03:35.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.220+0000 7fd5add9b700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:35.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.220+0000 7fd5add9b700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd59802ff60 0x7fd598032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd5a00096a0 tx=0x7fd5a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:35.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.224+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5a40079c0 con 0x7fd5b00f5410 2024-01-21T20:03:35.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.433+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5b00fae60 con 0x7fd5b00f5410 2024-01-21T20:03:35.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.433+0000 7fd59effd700 1 -- 172.21.15.153:0/1694988665 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd5a400f300 con 0x7fd5b00f5410 2024-01-21T20:03:35.424 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:35.424 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd59802ff60 msgr2=0x7fd598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd59802ff60 0x7fd598032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd5a00096a0 tx=0x7fd5a0006b40).stop 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 msgr2=0x7fd5b0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b0109430 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd5a4014000 tx=0x7fd5a400b040).stop 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 shutdown_connections 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd59802ff60 0x7fd598032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:35.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 --2- 172.21.15.153:0/1694988665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd5b00f5410 0x7fd5b0109430 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:35.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 >> 172.21.15.153:0/1694988665 conn(0x7fd5b00f0b70 msgr2=0x7fd5b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:35.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 shutdown_connections 2024-01-21T20:03:35.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:35.435+0000 7fd5b4fcb700 1 -- 172.21.15.153:0/1694988665 wait complete. 2024-01-21T20:03:35.426 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:36.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:34.682560+0000 mgr.smithi119.rqacbm (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:36.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:35.433789+0000 mon.smithi119 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.153:0/1694988665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:36.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:36.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:36.682800+0000 mgr.smithi119.rqacbm (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:38.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.021+0000 7f6d2c879700 1 -- 172.21.15.153:0/2310261344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 msgr2=0x7f6d280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:38.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.021+0000 7f6d2c879700 1 --2- 172.21.15.153:0/2310261344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d280f66c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6d18009fb0 tx=0x7f6d18009b30).stop 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 -- 172.21.15.153:0/2310261344 shutdown_connections 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 --2- 172.21.15.153:0/2310261344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d280f66c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 -- 172.21.15.153:0/2310261344 >> 172.21.15.153:0/2310261344 conn(0x7f6d280f1ac0 msgr2=0x7f6d280f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 -- 172.21.15.153:0/2310261344 shutdown_connections 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 -- 172.21.15.153:0/2310261344 wait complete. 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 Processor -- start 2024-01-21T20:03:38.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.022+0000 7f6d2c879700 1 -- start start 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d2c879700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d28106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d2c879700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d281069f0 con 0x7f6d280f62a0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d28106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d28106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56162/0 (socket says 172.21.15.153:56162) 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d277fe700 1 -- 172.21.15.153:0/1881607897 learned_addr learned my addr 172.21.15.153:0/1881607897 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d277fe700 1 -- 172.21.15.153:0/1881607897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18005040 con 0x7f6d280f62a0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d277fe700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d28106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6d18009830 tx=0x7f6d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d18009320 con 0x7f6d280f62a0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d28106bf0 con 0x7f6d280f62a0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d180136c0 con 0x7f6d280f62a0 2024-01-21T20:03:38.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.023+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d1801ee90 con 0x7f6d280f62a0 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.024+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d28107080 con 0x7f6d280f62a0 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.024+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6d1801e420 con 0x7f6d280f62a0 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.024+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d28045ad0 con 0x7f6d280f62a0 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.024+0000 7f6d0ffff700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6d1002ff10 0x7f6d100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.024+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6d18041050 con 0x7f6d280f62a0 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.025+0000 7f6d26ffd700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6d1002ff10 0x7f6d100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:38.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.025+0000 7f6d26ffd700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6d1002ff10 0x7f6d100323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6d1c0096a0 tx=0x7f6d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:38.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.029+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6d1806d2b0 con 0x7f6d280f62a0 2024-01-21T20:03:38.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.238+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d28107ed0 con 0x7f6d280f62a0 2024-01-21T20:03:38.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.239+0000 7f6d0ffff700 1 -- 172.21.15.153:0/1881607897 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6d180095f0 con 0x7f6d280f62a0 2024-01-21T20:03:38.230 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:38.230 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6d1002ff10 msgr2=0x7f6d100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6d1002ff10 0x7f6d100323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6d1c0096a0 tx=0x7f6d1c006b40).stop 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 msgr2=0x7f6d28106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d28106500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6d18009830 tx=0x7f6d1800b040).stop 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 shutdown_connections 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6d1002ff10 0x7f6d100323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:38.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 --2- 172.21.15.153:0/1881607897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d280f62a0 0x7f6d28106500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:38.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 >> 172.21.15.153:0/1881607897 conn(0x7f6d280f1ac0 msgr2=0x7f6d280f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:38.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 shutdown_connections 2024-01-21T20:03:38.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:38.241+0000 7f6d2c879700 1 -- 172.21.15.153:0/1881607897 wait complete. 2024-01-21T20:03:38.232 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:38.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:38.239726+0000 mon.smithi119 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.153:0/1881607897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:39.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:38.683137+0000 mgr.smithi119.rqacbm (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:39.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:39.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 -- 172.21.15.153:0/1873135842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 msgr2=0x7fb9340f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 --2- 172.21.15.153:0/1873135842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9340f4b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fb928009fb0 tx=0x7fb928009b30).stop 2024-01-21T20:03:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 -- 172.21.15.153:0/1873135842 shutdown_connections 2024-01-21T20:03:40.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 --2- 172.21.15.153:0/1873135842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9340f4b60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 -- 172.21.15.153:0/1873135842 >> 172.21.15.153:0/1873135842 conn(0x7fb9340f0260 msgr2=0x7fb9340f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 -- 172.21.15.153:0/1873135842 shutdown_connections 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.801+0000 7fb9388d0700 1 -- 172.21.15.153:0/1873135842 wait complete. 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb9388d0700 1 Processor -- start 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb9388d0700 1 -- start start 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb9388d0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9341066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb9388d0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb934106bc0 con 0x7fb9340f4740 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb93259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9341066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb93259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9341066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:56180/0 (socket says 172.21.15.153:56180) 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.802+0000 7fb93259c700 1 -- 172.21.15.153:0/2359696018 learned_addr learned my addr 172.21.15.153:0/2359696018 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb93259c700 1 -- 172.21.15.153:0/2359696018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928005040 con 0x7fb9340f4740 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb93259c700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9341066d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb928014000 tx=0x7fb92800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9280095f0 con 0x7fb9340f4740 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb934106dc0 con 0x7fb9340f4740 2024-01-21T20:03:40.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb928007e70 con 0x7fb9340f4740 2024-01-21T20:03:40.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb928014ea0 con 0x7fb9340f4740 2024-01-21T20:03:40.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.803+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb934107250 con 0x7fb9340f4740 2024-01-21T20:03:40.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.804+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb928025070 con 0x7fb9340f4740 2024-01-21T20:03:40.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.804+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb934045ad0 con 0x7fb9340f4740 2024-01-21T20:03:40.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.804+0000 7fb922ffd700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb91c02ff60 0x7fb91c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:40.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.804+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb928040a30 con 0x7fb9340f4740 2024-01-21T20:03:40.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.804+0000 7fb931d9b700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb91c02ff60 0x7fb91c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:40.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.804+0000 7fb931d9b700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb91c02ff60 0x7fb91c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb9240096a0 tx=0x7fb924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:40.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:40.808+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9280079c0 con 0x7fb9340f4740 2024-01-21T20:03:41.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.018+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb934108100 con 0x7fb9340f4740 2024-01-21T20:03:41.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.018+0000 7fb922ffd700 1 -- 172.21.15.153:0/2359696018 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb92800f300 con 0x7fb9340f4740 2024-01-21T20:03:41.009 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:41.009 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:41.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb91c02ff60 msgr2=0x7fb91c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:41.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb91c02ff60 0x7fb91c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb9240096a0 tx=0x7fb924006b40).stop 2024-01-21T20:03:41.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 msgr2=0x7fb9341066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9341066d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb928014000 tx=0x7fb92800b040).stop 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 shutdown_connections 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb91c02ff60 0x7fb91c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 --2- 172.21.15.153:0/2359696018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9340f4740 0x7fb9341066d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 >> 172.21.15.153:0/2359696018 conn(0x7fb9340f0260 msgr2=0x7fb9340f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 shutdown_connections 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:41.020+0000 7fb9388d0700 1 -- 172.21.15.153:0/2359696018 wait complete. 2024-01-21T20:03:41.011 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:41.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:40.683518+0000 mgr.smithi119.rqacbm (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:42.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:41.018868+0000 mon.smithi119 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.153:0/2359696018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:42.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:42.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:43.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:42.683830+0000 mgr.smithi119.rqacbm (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:43.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 -- 172.21.15.153:0/2126929026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 msgr2=0x7efdb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:43.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 --2- 172.21.15.153:0/2126929026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb00f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7efda4009fb0 tx=0x7efda4009b30).stop 2024-01-21T20:03:43.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 -- 172.21.15.153:0/2126929026 shutdown_connections 2024-01-21T20:03:43.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 --2- 172.21.15.153:0/2126929026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb00f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:43.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 -- 172.21.15.153:0/2126929026 >> 172.21.15.153:0/2126929026 conn(0x7efdb00f1c70 msgr2=0x7efdb00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:43.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 -- 172.21.15.153:0/2126929026 shutdown_connections 2024-01-21T20:03:43.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 -- 172.21.15.153:0/2126929026 wait complete. 2024-01-21T20:03:43.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.596+0000 7efdb646a700 1 Processor -- start 2024-01-21T20:03:43.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.597+0000 7efdb646a700 1 -- start start 2024-01-21T20:03:43.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.597+0000 7efdb646a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:43.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.597+0000 7efdb646a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0106c20 con 0x7efdb00f6510 2024-01-21T20:03:43.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.597+0000 7efdaffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:43.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.597+0000 7efdaffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40394/0 (socket says 172.21.15.153:40394) 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.597+0000 7efdaffff700 1 -- 172.21.15.153:0/3488383568 learned_addr learned my addr 172.21.15.153:0/3488383568 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdaffff700 1 -- 172.21.15.153:0/3488383568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4005040 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdaffff700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb0106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7efda4013510 tx=0x7efda400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda4009320 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb0106e20 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efda4013f00 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda4014d90 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.598+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb01072b0 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.599+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efda4014420 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.599+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb0045ad0 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.599+0000 7efdacff9700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efd9802ff60 0x7efd98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.599+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efda4041110 con 0x7efdb00f6510 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.599+0000 7efdaf7fe700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efd9802ff60 0x7efd98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:43.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.599+0000 7efdaf7fe700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efd9802ff60 0x7efd98032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efda00096a0 tx=0x7efda0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:43.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.603+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efda40095f0 con 0x7efdb00f6510 2024-01-21T20:03:43.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.811+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdb00fbf60 con 0x7efdb00f6510 2024-01-21T20:03:43.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.812+0000 7efdacff9700 1 -- 172.21.15.153:0/3488383568 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efda401db70 con 0x7efdb00f6510 2024-01-21T20:03:43.805 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:43.805 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:43.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efd9802ff60 msgr2=0x7efd98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:43.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efd9802ff60 0x7efd98032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efda00096a0 tx=0x7efda0006b40).stop 2024-01-21T20:03:43.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 msgr2=0x7efdb0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:43.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb0106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7efda4013510 tx=0x7efda400b040).stop 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 shutdown_connections 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7efd9802ff60 0x7efd98032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 --2- 172.21.15.153:0/3488383568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efdb00f6510 0x7efdb0106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.813+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 >> 172.21.15.153:0/3488383568 conn(0x7efdb00f1c70 msgr2=0x7efdb00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.814+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 shutdown_connections 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:43.814+0000 7efdb646a700 1 -- 172.21.15.153:0/3488383568 wait complete. 2024-01-21T20:03:43.806 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:44.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:43.813386+0000 mon.smithi119 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.153:0/3488383568' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:45.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:45.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:45.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:44.684176+0000 mgr.smithi119.rqacbm (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:46.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.435+0000 7f1cdf916700 1 -- 172.21.15.153:0/1541483364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 msgr2=0x7f1cd80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.435+0000 7f1cdf916700 1 --2- 172.21.15.153:0/1541483364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f4be0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f1ccc009fb0 tx=0x7f1ccc009b30).stop 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.435+0000 7f1cdf916700 1 -- 172.21.15.153:0/1541483364 shutdown_connections 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.435+0000 7f1cdf916700 1 --2- 172.21.15.153:0/1541483364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f4be0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.435+0000 7f1cdf916700 1 -- 172.21.15.153:0/1541483364 >> 172.21.15.153:0/1541483364 conn(0x7f1cd80f02e0 msgr2=0x7f1cd80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdf916700 1 -- 172.21.15.153:0/1541483364 shutdown_connections 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdf916700 1 -- 172.21.15.153:0/1541483364 wait complete. 2024-01-21T20:03:46.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdf916700 1 Processor -- start 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdf916700 1 -- start start 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdf916700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdf916700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd80f57f0 con 0x7f1cd80f47c0 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdd6b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdd6b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40418/0 (socket says 172.21.15.153:40418) 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.436+0000 7f1cdd6b2700 1 -- 172.21.15.153:0/3209752878 learned_addr learned my addr 172.21.15.153:0/3209752878 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cdd6b2700 1 -- 172.21.15.153:0/3209752878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc005040 con 0x7f1cd80f47c0 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cdd6b2700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f7370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1ccc013510 tx=0x7f1ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc009320 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd80f5990 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ccc013f00 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc014d90 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.437+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd80f5e20 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.438+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ccc014420 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.438+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd8059100 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.438+0000 7f1cc9ffb700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1cc402ff60 0x7f1cc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.438+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ccc041000 con 0x7f1cd80f47c0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.438+0000 7f1cdceb1700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1cc402ff60 0x7f1cc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.439+0000 7f1cdceb1700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1cc402ff60 0x7f1cc4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1cd40096a0 tx=0x7f1cd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:46.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.443+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ccc0095f0 con 0x7f1cd80f47c0 2024-01-21T20:03:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.647+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cd80fbf60 con 0x7f1cd80f47c0 2024-01-21T20:03:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.647+0000 7f1cc9ffb700 1 -- 172.21.15.153:0/3209752878 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ccc01db70 con 0x7f1cd80f47c0 2024-01-21T20:03:46.639 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:46.639 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1cc402ff60 msgr2=0x7f1cc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1cc402ff60 0x7f1cc4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1cd40096a0 tx=0x7f1cd4006b40).stop 2024-01-21T20:03:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 msgr2=0x7f1cd80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f7370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1ccc013510 tx=0x7f1ccc00b040).stop 2024-01-21T20:03:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 shutdown_connections 2024-01-21T20:03:46.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1cc402ff60 0x7f1cc4032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 --2- 172.21.15.153:0/3209752878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cd80f47c0 0x7f1cd80f7370 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 >> 172.21.15.153:0/3209752878 conn(0x7f1cd80f02e0 msgr2=0x7f1cd80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 shutdown_connections 2024-01-21T20:03:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:46.649+0000 7f1cdf916700 1 -- 172.21.15.153:0/3209752878 wait complete. 2024-01-21T20:03:46.641 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:46.648662+0000 mon.smithi119 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.153:0/3209752878' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:46.684567+0000 mgr.smithi119.rqacbm (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:48.038 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:48.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:49.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:48.684782+0000 mgr.smithi119.rqacbm (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 -- 172.21.15.153:0/1094674214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 msgr2=0x7f74a00f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 --2- 172.21.15.153:0/1094674214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a00f7f80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f7498009960 tx=0x7f7498004d10).stop 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 -- 172.21.15.153:0/1094674214 shutdown_connections 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 --2- 172.21.15.153:0/1094674214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a00f7f80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 -- 172.21.15.153:0/1094674214 >> 172.21.15.153:0/1094674214 conn(0x7f74a00f02e0 msgr2=0x7f74a00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 -- 172.21.15.153:0/1094674214 shutdown_connections 2024-01-21T20:03:49.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.265+0000 7f74a86d9700 1 -- 172.21.15.153:0/1094674214 wait complete. 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a86d9700 1 Processor -- start 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a86d9700 1 -- start start 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a86d9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a0106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a86d9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0106c30 con 0x7f74a00f7b60 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a6475700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a0106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a6475700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a0106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40422/0 (socket says 172.21.15.153:40422) 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a6475700 1 -- 172.21.15.153:0/2888785478 learned_addr learned my addr 172.21.15.153:0/2888785478 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.266+0000 7f74a6475700 1 -- 172.21.15.153:0/2888785478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498009640 con 0x7f74a00f7b60 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.267+0000 7f74a6475700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a0106740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7498004b70 tx=0x7f7498004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.267+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7498018070 con 0x7f74a00f7b60 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.267+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a0106e30 con 0x7f74a00f7b60 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.267+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f749800b870 con 0x7f74a00f7b60 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.267+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f749801c560 con 0x7f74a00f7b60 2024-01-21T20:03:49.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.267+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a01072c0 con 0x7f74a00f7b60 2024-01-21T20:03:49.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.268+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7498013630 con 0x7f74a00f7b60 2024-01-21T20:03:49.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.268+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a0045ad0 con 0x7f74a00f7b60 2024-01-21T20:03:49.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.268+0000 7f749effd700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f748802ff60 0x7f7488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:49.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.268+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74980411d0 con 0x7f74a00f7b60 2024-01-21T20:03:49.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.268+0000 7f74a5c74700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f748802ff60 0x7f7488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:49.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.268+0000 7f74a5c74700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f748802ff60 0x7f7488032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f74940096a0 tx=0x7f7494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:49.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.272+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7498004810 con 0x7f74a00f7b60 2024-01-21T20:03:49.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.476+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74a0108110 con 0x7f74a00f7b60 2024-01-21T20:03:49.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.477+0000 7f749effd700 1 -- 172.21.15.153:0/2888785478 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7498012380 con 0x7f74a00f7b60 2024-01-21T20:03:49.468 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:49.468 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f748802ff60 msgr2=0x7f7488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f748802ff60 0x7f7488032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f74940096a0 tx=0x7f7494006b40).stop 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 msgr2=0x7f74a0106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a0106740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7498004b70 tx=0x7f7498004d10).stop 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 shutdown_connections 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f748802ff60 0x7f7488032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 --2- 172.21.15.153:0/2888785478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f74a00f7b60 0x7f74a0106740 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 >> 172.21.15.153:0/2888785478 conn(0x7f74a00f02e0 msgr2=0x7f74a00f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 shutdown_connections 2024-01-21T20:03:49.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:49.479+0000 7f74a86d9700 1 -- 172.21.15.153:0/2888785478 wait complete. 2024-01-21T20:03:49.471 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:50.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:49.478446+0000 mon.smithi119 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.153:0/2888785478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:50.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:50.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:50.685089+0000 mgr.smithi119.rqacbm (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 -- 172.21.15.153:0/2036380303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 msgr2=0x7fada80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 --2- 172.21.15.153:0/2036380303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada80f6930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fad98009fb0 tx=0x7fad98009b30).stop 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 -- 172.21.15.153:0/2036380303 shutdown_connections 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 --2- 172.21.15.153:0/2036380303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada80f6930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 -- 172.21.15.153:0/2036380303 >> 172.21.15.153:0/2036380303 conn(0x7fada80f1c70 msgr2=0x7fada80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 -- 172.21.15.153:0/2036380303 shutdown_connections 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.106+0000 7fadaca87700 1 -- 172.21.15.153:0/2036380303 wait complete. 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fadaca87700 1 Processor -- start 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fadaca87700 1 -- start start 2024-01-21T20:03:52.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fadaca87700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:52.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fadaca87700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada8106c20 con 0x7fada80f6510 2024-01-21T20:03:52.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fada659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:52.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fada659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40430/0 (socket says 172.21.15.153:40430) 2024-01-21T20:03:52.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.107+0000 7fada659c700 1 -- 172.21.15.153:0/3014811854 learned_addr learned my addr 172.21.15.153:0/3014811854 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:52.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fada659c700 1 -- 172.21.15.153:0/3014811854 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad98005040 con 0x7fada80f6510 2024-01-21T20:03:52.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fada659c700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada8106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fad98014000 tx=0x7fad9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:52.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad980095f0 con 0x7fada80f6510 2024-01-21T20:03:52.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada8106e20 con 0x7fada80f6510 2024-01-21T20:03:52.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad98007e70 con 0x7fada80f6510 2024-01-21T20:03:52.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad98014ea0 con 0x7fada80f6510 2024-01-21T20:03:52.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.108+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada81072b0 con 0x7fada80f6510 2024-01-21T20:03:52.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.109+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad98025070 con 0x7fada80f6510 2024-01-21T20:03:52.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.109+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada8045ad0 con 0x7fada80f6510 2024-01-21T20:03:52.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.109+0000 7fad9effd700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fad9402ff60 0x7fad94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:52.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.109+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad98040950 con 0x7fada80f6510 2024-01-21T20:03:52.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.109+0000 7fada5d9b700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fad9402ff60 0x7fad94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:52.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.110+0000 7fada5d9b700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fad9402ff60 0x7fad94032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fad900096a0 tx=0x7fad90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:52.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.114+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad9800f4e0 con 0x7fada80f6510 2024-01-21T20:03:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.320+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fada80fbf60 con 0x7fada80f6510 2024-01-21T20:03:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.320+0000 7fad9effd700 1 -- 172.21.15.153:0/3014811854 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fad9800f300 con 0x7fada80f6510 2024-01-21T20:03:52.311 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:52.312 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fad9402ff60 msgr2=0x7fad94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fad9402ff60 0x7fad94032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fad900096a0 tx=0x7fad90006b40).stop 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 msgr2=0x7fada8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada8106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fad98014000 tx=0x7fad9800b040).stop 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 shutdown_connections 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fad9402ff60 0x7fad94032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 --2- 172.21.15.153:0/3014811854 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fada80f6510 0x7fada8106730 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:52.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 >> 172.21.15.153:0/3014811854 conn(0x7fada80f1c70 msgr2=0x7fada80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 shutdown_connections 2024-01-21T20:03:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:52.322+0000 7fadaca87700 1 -- 172.21.15.153:0/3014811854 wait complete. 2024-01-21T20:03:52.314 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:52.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:52.321836+0000 mon.smithi119 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.153:0/3014811854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:53.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:53.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:53.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:52.685451+0000 mgr.smithi119.rqacbm (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 -- 172.21.15.153:0/3648021851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 msgr2=0x7f1d340f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 --2- 172.21.15.153:0/3648021851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d340f4be0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1d30009fb0 tx=0x7f1d30009b30).stop 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 -- 172.21.15.153:0/3648021851 shutdown_connections 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 --2- 172.21.15.153:0/3648021851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d340f4be0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 -- 172.21.15.153:0/3648021851 >> 172.21.15.153:0/3648021851 conn(0x7f1d340f02e0 msgr2=0x7f1d340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 -- 172.21.15.153:0/3648021851 shutdown_connections 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 -- 172.21.15.153:0/3648021851 wait complete. 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 Processor -- start 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.914+0000 7f1d3b746700 1 -- start start 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d3b746700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d34106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:54.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d3b746700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34106bf0 con 0x7f1d340f47c0 2024-01-21T20:03:54.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d394e2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d34106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:54.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d394e2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d34106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:60084/0 (socket says 172.21.15.153:60084) 2024-01-21T20:03:54.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d394e2700 1 -- 172.21.15.153:0/1739760783 learned_addr learned my addr 172.21.15.153:0/1739760783 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:54.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d394e2700 1 -- 172.21.15.153:0/1739760783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d30005040 con 0x7f1d340f47c0 2024-01-21T20:03:54.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d394e2700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d34106700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1d30013510 tx=0x7f1d3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:54.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d30009320 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d34106df0 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d30013f00 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.915+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d30014d90 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.916+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d34107280 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.916+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d34045ad0 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.916+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d300095f0 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.917+0000 7f1d29ffb700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d2002ff60 0x7f1d20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.917+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d30041000 con 0x7f1d340f47c0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.917+0000 7f1d38ce1700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d2002ff60 0x7f1d20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:54.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.917+0000 7f1d38ce1700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d2002ff60 0x7f1d20032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1d240096a0 tx=0x7f1d24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:54.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:54.921+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d3001d7f0 con 0x7f1d340f47c0 2024-01-21T20:03:55.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:54.685721+0000 mgr.smithi119.rqacbm (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:55.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.127+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d34108130 con 0x7f1d340f47c0 2024-01-21T20:03:55.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.127+0000 7f1d29ffb700 1 -- 172.21.15.153:0/1739760783 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d30010230 con 0x7f1d340f47c0 2024-01-21T20:03:55.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:55.119 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d2002ff60 msgr2=0x7f1d20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d2002ff60 0x7f1d20032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1d240096a0 tx=0x7f1d24006b40).stop 2024-01-21T20:03:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 msgr2=0x7f1d34106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d34106700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1d30013510 tx=0x7f1d3000b040).stop 2024-01-21T20:03:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 shutdown_connections 2024-01-21T20:03:55.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1d2002ff60 0x7f1d20032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 --2- 172.21.15.153:0/1739760783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d340f47c0 0x7f1d34106700 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 >> 172.21.15.153:0/1739760783 conn(0x7f1d340f02e0 msgr2=0x7f1d340f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 shutdown_connections 2024-01-21T20:03:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:55.129+0000 7f1d3b746700 1 -- 172.21.15.153:0/1739760783 wait complete. 2024-01-21T20:03:55.121 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:56.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:55 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:55.128698+0000 mon.smithi119 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.153:0/1739760783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:56.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:56.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:56.686081+0000 mgr.smithi119.rqacbm (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1107238566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 msgr2=0x7ff9f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 --2- 172.21.15.153:0/1107238566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f00f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff9d8009fb0 tx=0x7ff9d8009b30).stop 2024-01-21T20:03:57.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1107238566 shutdown_connections 2024-01-21T20:03:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 --2- 172.21.15.153:0/1107238566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f00f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1107238566 >> 172.21.15.153:0/1107238566 conn(0x7ff9f00f1c70 msgr2=0x7ff9f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1107238566 shutdown_connections 2024-01-21T20:03:57.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.671+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1107238566 wait complete. 2024-01-21T20:03:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9f50b8700 1 Processor -- start 2024-01-21T20:03:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9f50b8700 1 -- start start 2024-01-21T20:03:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9f50b8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9f50b8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f0106c20 con 0x7ff9f00f6510 2024-01-21T20:03:57.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9eed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9eed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:60104/0 (socket says 172.21.15.153:60104) 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.672+0000 7ff9eed9d700 1 -- 172.21.15.153:0/1928410088 learned_addr learned my addr 172.21.15.153:0/1928410088 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9eed9d700 1 -- 172.21.15.153:0/1928410088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8005040 con 0x7ff9f00f6510 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9eed9d700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f0106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff9d8013510 tx=0x7ff9d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8009320 con 0x7ff9f00f6510 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f0106e20 con 0x7ff9f00f6510 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9d8013f00 con 0x7ff9f00f6510 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d8014d90 con 0x7ff9f00f6510 2024-01-21T20:03:57.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.673+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f01072b0 con 0x7ff9f00f6510 2024-01-21T20:03:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.674+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9d80095f0 con 0x7ff9f00f6510 2024-01-21T20:03:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.674+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f0045ad0 con 0x7ff9f00f6510 2024-01-21T20:03:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.674+0000 7ff9e77fe700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9dc02ff60 0x7ff9dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:03:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.674+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9d80410c0 con 0x7ff9f00f6510 2024-01-21T20:03:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.674+0000 7ff9ee59c700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9dc02ff60 0x7ff9dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:03:57.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.674+0000 7ff9ee59c700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9dc02ff60 0x7ff9dc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff9e00096a0 tx=0x7ff9e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:03:57.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.678+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9d8014420 con 0x7ff9f00f6510 2024-01-21T20:03:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.885+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9f00fbf60 con 0x7ff9f00f6510 2024-01-21T20:03:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.885+0000 7ff9e77fe700 1 -- 172.21.15.153:0/1928410088 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff9d801d7f0 con 0x7ff9f00f6510 2024-01-21T20:03:57.877 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:03:57.877 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9dc02ff60 msgr2=0x7ff9dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9dc02ff60 0x7ff9dc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff9e00096a0 tx=0x7ff9e0006b40).stop 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 msgr2=0x7ff9f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f0106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff9d8013510 tx=0x7ff9d800b040).stop 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 shutdown_connections 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9dc02ff60 0x7ff9dc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 --2- 172.21.15.153:0/1928410088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f00f6510 0x7ff9f0106730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:03:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 >> 172.21.15.153:0/1928410088 conn(0x7ff9f00f1c70 msgr2=0x7ff9f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:03:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 shutdown_connections 2024-01-21T20:03:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:03:57.887+0000 7ff9f50b8700 1 -- 172.21.15.153:0/1928410088 wait complete. 2024-01-21T20:03:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:03:58.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:57.886901+0000 mon.smithi119 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.153:0/1928410088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:03:59.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:03:59.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:03:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:03:58.686434+0000 mgr.smithi119.rqacbm (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:03:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:58.783435+0000 mon.smithi119 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:03:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:03:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:03:58.783826+0000 mon.smithi119 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:04:00.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 -- 172.21.15.153:0/1871151258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 msgr2=0x7f7f880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 --2- 172.21.15.153:0/1871151258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f880f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f7f7c009fb0 tx=0x7f7f7c009b30).stop 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 -- 172.21.15.153:0/1871151258 shutdown_connections 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 --2- 172.21.15.153:0/1871151258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f880f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 -- 172.21.15.153:0/1871151258 >> 172.21.15.153:0/1871151258 conn(0x7f7f880f0b70 msgr2=0x7f7f880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 -- 172.21.15.153:0/1871151258 shutdown_connections 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 -- 172.21.15.153:0/1871151258 wait complete. 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.431+0000 7f7f8f075700 1 Processor -- start 2024-01-21T20:04:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8f075700 1 -- start start 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8f075700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f881093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8f075700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f88060220 con 0x7f7f880f5410 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8ce11700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f881093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8ce11700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f881093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:60120/0 (socket says 172.21.15.153:60120) 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8ce11700 1 -- 172.21.15.153:0/99264808 learned_addr learned my addr 172.21.15.153:0/99264808 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8ce11700 1 -- 172.21.15.153:0/99264808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c005040 con 0x7f7f880f5410 2024-01-21T20:04:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.432+0000 7f7f8ce11700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f881093f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f7f7c013510 tx=0x7f7f7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.433+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f7c009320 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.433+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f88060420 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.433+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f7c013f00 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.433+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f7c014d90 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.433+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f88109aa0 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.433+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f7c0263a0 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.434+0000 7f7f857fa700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f7802ff60 0x7f7f78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.434+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f7c040ff0 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.434+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f88059100 con 0x7f7f880f5410 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.434+0000 7f7f87fff700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f7802ff60 0x7f7f78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:00.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.434+0000 7f7f87fff700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f7802ff60 0x7f7f78032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7f740096a0 tx=0x7f7f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:00.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.439+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f7c012920 con 0x7f7f880f5410 2024-01-21T20:04:00.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.645+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f88045ad0 con 0x7f7f880f5410 2024-01-21T20:04:00.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.645+0000 7f7f857fa700 1 -- 172.21.15.153:0/99264808 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f7c02b020 con 0x7f7f880f5410 2024-01-21T20:04:00.636 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:00.637 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f7802ff60 msgr2=0x7f7f78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f7802ff60 0x7f7f78032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f7f740096a0 tx=0x7f7f74006b40).stop 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 msgr2=0x7f7f881093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f881093f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f7f7c013510 tx=0x7f7f7c00b040).stop 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 shutdown_connections 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f7802ff60 0x7f7f78032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 --2- 172.21.15.153:0/99264808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f880f5410 0x7f7f881093f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 >> 172.21.15.153:0/99264808 conn(0x7f7f880f0b70 msgr2=0x7f7f880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:00.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 shutdown_connections 2024-01-21T20:04:00.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:00.647+0000 7f7f8f075700 1 -- 172.21.15.153:0/99264808 wait complete. 2024-01-21T20:04:00.639 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:01.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:00.646744+0000 mon.smithi119 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.153:0/99264808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:02.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:02.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:02.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:01 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:00.686783+0000 mgr.smithi119.rqacbm (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:03.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 -- 172.21.15.153:0/3540044869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 msgr2=0x7f3f380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 --2- 172.21.15.153:0/3540044869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f380f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f3f28009fb0 tx=0x7f3f28009b30).stop 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 -- 172.21.15.153:0/3540044869 shutdown_connections 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 --2- 172.21.15.153:0/3540044869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f380f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 -- 172.21.15.153:0/3540044869 >> 172.21.15.153:0/3540044869 conn(0x7f3f380f0b70 msgr2=0x7f3f380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 -- 172.21.15.153:0/3540044869 shutdown_connections 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 -- 172.21.15.153:0/3540044869 wait complete. 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 Processor -- start 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.267+0000 7f3f3d941700 1 -- start start 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f3d941700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f381066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f3d941700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f38106bc0 con 0x7f3f380f5410 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f36ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f381066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f36ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f381066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:60150/0 (socket says 172.21.15.153:60150) 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f36ffd700 1 -- 172.21.15.153:0/3203056387 learned_addr learned my addr 172.21.15.153:0/3203056387 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f36ffd700 1 -- 172.21.15.153:0/3203056387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f28005040 con 0x7f3f380f5410 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f36ffd700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f381066d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3f28013510 tx=0x7f3f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f28009320 con 0x7f3f380f5410 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f38106dc0 con 0x7f3f380f5410 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f28013f00 con 0x7f3f380f5410 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f28014d90 con 0x7f3f380f5410 2024-01-21T20:04:03.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.268+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f38107250 con 0x7f3f380f5410 2024-01-21T20:04:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.269+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f28014420 con 0x7f3f380f5410 2024-01-21T20:04:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.269+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f38045ad0 con 0x7f3f380f5410 2024-01-21T20:04:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.269+0000 7f3f1f7fe700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3f2002ff60 0x7f3f20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.269+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f28041070 con 0x7f3f380f5410 2024-01-21T20:04:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.270+0000 7f3f367fc700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3f2002ff60 0x7f3f20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:03.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.270+0000 7f3f367fc700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3f2002ff60 0x7f3f20032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3f2c0096a0 tx=0x7f3f2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:03.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.274+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f280095f0 con 0x7f3f380f5410 2024-01-21T20:04:03.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.478+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f38108100 con 0x7f3f380f5410 2024-01-21T20:04:03.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.479+0000 7f3f1f7fe700 1 -- 172.21.15.153:0/3203056387 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f2801db70 con 0x7f3f380f5410 2024-01-21T20:04:03.470 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:03.470 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:03.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.480+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3f2002ff60 msgr2=0x7f3f20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.480+0000 7f3f3d941700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3f2002ff60 0x7f3f20032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3f2c0096a0 tx=0x7f3f2c006b40).stop 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.480+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 msgr2=0x7f3f381066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.480+0000 7f3f3d941700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f381066d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3f28013510 tx=0x7f3f2800b040).stop 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.481+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 shutdown_connections 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.481+0000 7f3f3d941700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3f2002ff60 0x7f3f20032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.481+0000 7f3f3d941700 1 --2- 172.21.15.153:0/3203056387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f380f5410 0x7f3f381066d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.481+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 >> 172.21.15.153:0/3203056387 conn(0x7f3f380f0b70 msgr2=0x7f3f3805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.481+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 shutdown_connections 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:03.481+0000 7f3f3d941700 1 -- 172.21.15.153:0/3203056387 wait complete. 2024-01-21T20:04:03.472 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:04.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:02.687085+0000 mgr.smithi119.rqacbm (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:04.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:03.480566+0000 mon.smithi119 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.153:0/3203056387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:04.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:04.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:06.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:04.687333+0000 mgr.smithi119.rqacbm (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:06.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 -- 172.21.15.153:0/271285966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 msgr2=0x7fd4f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 --2- 172.21.15.153:0/271285966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f00f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fd4e4009fb0 tx=0x7fd4e4009b30).stop 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 -- 172.21.15.153:0/271285966 shutdown_connections 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 --2- 172.21.15.153:0/271285966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f00f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 -- 172.21.15.153:0/271285966 >> 172.21.15.153:0/271285966 conn(0x7fd4f00f1c70 msgr2=0x7fd4f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 -- 172.21.15.153:0/271285966 shutdown_connections 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.124+0000 7fd4f54b4700 1 -- 172.21.15.153:0/271285966 wait complete. 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.125+0000 7fd4f54b4700 1 Processor -- start 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.125+0000 7fd4f54b4700 1 -- start start 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.125+0000 7fd4f54b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.125+0000 7fd4f54b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0106c20 con 0x7fd4f00f6510 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4eeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4eeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40074/0 (socket says 172.21.15.153:40074) 2024-01-21T20:04:06.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4eeffd700 1 -- 172.21.15.153:0/4178945056 learned_addr learned my addr 172.21.15.153:0/4178945056 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4eeffd700 1 -- 172.21.15.153:0/4178945056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4005040 con 0x7fd4f00f6510 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4eeffd700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f0106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fd4e4013510 tx=0x7fd4e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e4009320 con 0x7fd4f00f6510 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4e4013f00 con 0x7fd4f00f6510 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f0106e20 con 0x7fd4f00f6510 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e4014d90 con 0x7fd4f00f6510 2024-01-21T20:04:06.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.126+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f01072b0 con 0x7fd4f00f6510 2024-01-21T20:04:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.127+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4e4014420 con 0x7fd4f00f6510 2024-01-21T20:04:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.127+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f0045ad0 con 0x7fd4f00f6510 2024-01-21T20:04:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.127+0000 7fd4d77fe700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4d802ff60 0x7fd4d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.127+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4e40410c0 con 0x7fd4f00f6510 2024-01-21T20:04:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.127+0000 7fd4ee7fc700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4d802ff60 0x7fd4d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:06.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.128+0000 7fd4ee7fc700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4d802ff60 0x7fd4d8032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd4e00096a0 tx=0x7fd4e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:06.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.132+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4e40095f0 con 0x7fd4f00f6510 2024-01-21T20:04:06.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.338+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4f00fbf60 con 0x7fd4f00f6510 2024-01-21T20:04:06.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.339+0000 7fd4d77fe700 1 -- 172.21.15.153:0/4178945056 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd4e401db70 con 0x7fd4f00f6510 2024-01-21T20:04:06.330 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:06.330 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4d802ff60 msgr2=0x7fd4d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4d802ff60 0x7fd4d8032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd4e00096a0 tx=0x7fd4e0006b40).stop 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 msgr2=0x7fd4f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f0106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fd4e4013510 tx=0x7fd4e400b040).stop 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 shutdown_connections 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd4d802ff60 0x7fd4d8032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 --2- 172.21.15.153:0/4178945056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4f00f6510 0x7fd4f0106730 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:06.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 >> 172.21.15.153:0/4178945056 conn(0x7fd4f00f1c70 msgr2=0x7fd4f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:06.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 shutdown_connections 2024-01-21T20:04:06.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:06.341+0000 7fd4f54b4700 1 -- 172.21.15.153:0/4178945056 wait complete. 2024-01-21T20:04:06.333 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:06.340547+0000 mon.smithi119 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.153:0/4178945056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:06.687605+0000 mgr.smithi119.rqacbm (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:07.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:07.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 -- 172.21.15.153:0/3101559935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 msgr2=0x7fbdac0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 --2- 172.21.15.153:0/3101559935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f8f80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fbda0009fb0 tx=0x7fbda0009b30).stop 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 -- 172.21.15.153:0/3101559935 shutdown_connections 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 --2- 172.21.15.153:0/3101559935 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f8f80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 -- 172.21.15.153:0/3101559935 >> 172.21.15.153:0/3101559935 conn(0x7fbdac0f09a0 msgr2=0x7fbdac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 -- 172.21.15.153:0/3101559935 shutdown_connections 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 -- 172.21.15.153:0/3101559935 wait complete. 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 Processor -- start 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.956+0000 7fbdb1973700 1 -- start start 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb1973700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:08.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb1973700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac0f4e80 con 0x7fbdac060180 2024-01-21T20:04:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb0971700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb0971700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40100/0 (socket says 172.21.15.153:40100) 2024-01-21T20:04:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb0971700 1 -- 172.21.15.153:0/2958754718 learned_addr learned my addr 172.21.15.153:0/2958754718 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:08.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb0971700 1 -- 172.21.15.153:0/2958754718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0005040 con 0x7fbdac060180 2024-01-21T20:04:08.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.957+0000 7fbdb0971700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f6a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbda0013510 tx=0x7fbda000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:08.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.958+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbda0009320 con 0x7fbdac060180 2024-01-21T20:04:08.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.958+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbda0013f00 con 0x7fbdac060180 2024-01-21T20:04:08.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.958+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac0f5020 con 0x7fbdac060180 2024-01-21T20:04:08.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.958+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbda0014d90 con 0x7fbdac060180 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.958+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac0f54b0 con 0x7fbdac060180 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.958+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbda00095f0 con 0x7fbdac060180 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.959+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdac059100 con 0x7fbdac060180 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.959+0000 7fbda97fa700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbd9c02ff60 0x7fbd9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.959+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbda00410f0 con 0x7fbdac060180 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.959+0000 7fbdabfff700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbd9c02ff60 0x7fbd9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:08.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.959+0000 7fbdabfff700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbd9c02ff60 0x7fbd9c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbd980096a0 tx=0x7fbd98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:08.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:08.963+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbda00268c0 con 0x7fbdac060180 2024-01-21T20:04:09.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:08.687898+0000 mgr.smithi119.rqacbm (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:09.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.170+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdac0fbce0 con 0x7fbdac060180 2024-01-21T20:04:09.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.170+0000 7fbda97fa700 1 -- 172.21.15.153:0/2958754718 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbda0010230 con 0x7fbdac060180 2024-01-21T20:04:09.165 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:09.165 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:09.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbd9c02ff60 msgr2=0x7fbd9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:09.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbd9c02ff60 0x7fbd9c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fbd980096a0 tx=0x7fbd98006b40).stop 2024-01-21T20:04:09.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 msgr2=0x7fbdac0f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f6a00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbda0013510 tx=0x7fbda000b040).stop 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 shutdown_connections 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbd9c02ff60 0x7fbd9c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 --2- 172.21.15.153:0/2958754718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdac060180 0x7fbdac0f6a00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 >> 172.21.15.153:0/2958754718 conn(0x7fbdac0f09a0 msgr2=0x7fbdac0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 shutdown_connections 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:09.172+0000 7fbdb1973700 1 -- 172.21.15.153:0/2958754718 wait complete. 2024-01-21T20:04:09.166 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:10.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:09.171793+0000 mon.smithi119 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.153:0/2958754718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:10.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:10.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:10.688202+0000 mgr.smithi119.rqacbm (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 -- 172.21.15.153:0/602537458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 msgr2=0x7fea5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 --2- 172.21.15.153:0/602537458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c0f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fea50009fb0 tx=0x7fea50009b30).stop 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 -- 172.21.15.153:0/602537458 shutdown_connections 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 --2- 172.21.15.153:0/602537458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c0f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 -- 172.21.15.153:0/602537458 >> 172.21.15.153:0/602537458 conn(0x7fea5c0f0b70 msgr2=0x7fea5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 -- 172.21.15.153:0/602537458 shutdown_connections 2024-01-21T20:04:11.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.753+0000 7fea62311700 1 -- 172.21.15.153:0/602537458 wait complete. 2024-01-21T20:04:11.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea62311700 1 Processor -- start 2024-01-21T20:04:11.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea62311700 1 -- start start 2024-01-21T20:04:11.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea62311700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:11.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea62311700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c106c00 con 0x7fea5c0f5410 2024-01-21T20:04:11.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea5bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:11.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea5bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40124/0 (socket says 172.21.15.153:40124) 2024-01-21T20:04:11.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.754+0000 7fea5bfff700 1 -- 172.21.15.153:0/1187632267 learned_addr learned my addr 172.21.15.153:0/1187632267 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:11.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea5bfff700 1 -- 172.21.15.153:0/1187632267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea50005040 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea5bfff700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c106710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fea50014000 tx=0x7fea5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea500095f0 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea50007e70 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c106e00 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea50014ea0 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.755+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c107290 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.756+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea50025070 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.756+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c045ad0 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.756+0000 7fea58ff9700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea4402ff60 0x7fea44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.756+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea50040a30 con 0x7fea5c0f5410 2024-01-21T20:04:11.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.756+0000 7fea5b7fe700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea4402ff60 0x7fea44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:11.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.757+0000 7fea5b7fe700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea4402ff60 0x7fea44032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fea4c0096a0 tx=0x7fea4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:11.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.761+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea500079c0 con 0x7fea5c0f5410 2024-01-21T20:04:11.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.970+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea5c0fae60 con 0x7fea5c0f5410 2024-01-21T20:04:11.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.970+0000 7fea58ff9700 1 -- 172.21.15.153:0/1187632267 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea5000f300 con 0x7fea5c0f5410 2024-01-21T20:04:11.962 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:11.962 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea4402ff60 msgr2=0x7fea44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea4402ff60 0x7fea44032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fea4c0096a0 tx=0x7fea4c006b40).stop 2024-01-21T20:04:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 msgr2=0x7fea5c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c106710 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fea50014000 tx=0x7fea5000b040).stop 2024-01-21T20:04:11.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 shutdown_connections 2024-01-21T20:04:11.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fea4402ff60 0x7fea44032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:11.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 --2- 172.21.15.153:0/1187632267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea5c0f5410 0x7fea5c106710 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:11.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 >> 172.21.15.153:0/1187632267 conn(0x7fea5c0f0b70 msgr2=0x7fea5c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:11.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 shutdown_connections 2024-01-21T20:04:11.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:11.972+0000 7fea62311700 1 -- 172.21.15.153:0/1187632267 wait complete. 2024-01-21T20:04:11.965 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:12.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:11.972092+0000 mon.smithi119 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.153:0/1187632267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:13.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:13.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:13.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:12.688552+0000 mgr.smithi119.rqacbm (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:14.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 -- 172.21.15.153:0/2361250822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 msgr2=0x7feef00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:14.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 --2- 172.21.15.153:0/2361250822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef00f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7feed8009fb0 tx=0x7feed8009b30).stop 2024-01-21T20:04:14.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 -- 172.21.15.153:0/2361250822 shutdown_connections 2024-01-21T20:04:14.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 --2- 172.21.15.153:0/2361250822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef00f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:14.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 -- 172.21.15.153:0/2361250822 >> 172.21.15.153:0/2361250822 conn(0x7feef00f0b70 msgr2=0x7feef00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:14.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 -- 172.21.15.153:0/2361250822 shutdown_connections 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.519+0000 7feef4cb2700 1 -- 172.21.15.153:0/2361250822 wait complete. 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feef4cb2700 1 Processor -- start 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feef4cb2700 1 -- start start 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feef4cb2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feef4cb2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef0106bd0 con 0x7feef00f5410 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feeee59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feeee59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:42766/0 (socket says 172.21.15.153:42766) 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.520+0000 7feeee59c700 1 -- 172.21.15.153:0/448006441 learned_addr learned my addr 172.21.15.153:0/448006441 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feeee59c700 1 -- 172.21.15.153:0/448006441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed8005040 con 0x7feef00f5410 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feeee59c700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef01066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7feed8013510 tx=0x7feed800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feed8009320 con 0x7feef00f5410 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef0106dd0 con 0x7feef00f5410 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feed8013f00 con 0x7feef00f5410 2024-01-21T20:04:14.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feed8014d90 con 0x7feef00f5410 2024-01-21T20:04:14.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.521+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef0107260 con 0x7feef00f5410 2024-01-21T20:04:14.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.522+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feed80263a0 con 0x7feef00f5410 2024-01-21T20:04:14.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.522+0000 7feee6ffd700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7feedc02ff60 0x7feedc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:14.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.522+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feed80410b0 con 0x7feef00f5410 2024-01-21T20:04:14.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.522+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0045ad0 con 0x7feef00f5410 2024-01-21T20:04:14.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.522+0000 7feeedd9b700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7feedc02ff60 0x7feedc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:14.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.522+0000 7feeedd9b700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7feedc02ff60 0x7feedc032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7feee00096a0 tx=0x7feee0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:14.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.527+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feed8012920 con 0x7feef00f5410 2024-01-21T20:04:14.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.730+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feef0107fb0 con 0x7feef00f5410 2024-01-21T20:04:14.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.731+0000 7feee6ffd700 1 -- 172.21.15.153:0/448006441 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feed802b020 con 0x7feef00f5410 2024-01-21T20:04:14.722 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:14.722 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7feedc02ff60 msgr2=0x7feedc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7feedc02ff60 0x7feedc032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7feee00096a0 tx=0x7feee0006b40).stop 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 msgr2=0x7feef01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef01066e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7feed8013510 tx=0x7feed800b040).stop 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 shutdown_connections 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7feedc02ff60 0x7feedc032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 --2- 172.21.15.153:0/448006441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feef00f5410 0x7feef01066e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 >> 172.21.15.153:0/448006441 conn(0x7feef00f0b70 msgr2=0x7feef005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 shutdown_connections 2024-01-21T20:04:14.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:14.733+0000 7feef4cb2700 1 -- 172.21.15.153:0/448006441 wait complete. 2024-01-21T20:04:14.725 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:14.688893+0000 mgr.smithi119.rqacbm (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:14.732651+0000 mon.smithi119 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.153:0/448006441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:16.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:16.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:17.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 -- 172.21.15.153:0/4246711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 msgr2=0x7fd1640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 --2- 172.21.15.153:0/4246711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1640f55c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fd154009fb0 tx=0x7fd154009b30).stop 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 -- 172.21.15.153:0/4246711 shutdown_connections 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 --2- 172.21.15.153:0/4246711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1640f55c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 -- 172.21.15.153:0/4246711 >> 172.21.15.153:0/4246711 conn(0x7fd1640f09a0 msgr2=0x7fd1640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 -- 172.21.15.153:0/4246711 shutdown_connections 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 -- 172.21.15.153:0/4246711 wait complete. 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 Processor -- start 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.356+0000 7fd168b3a700 1 -- start start 2024-01-21T20:04:17.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd168b3a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1641053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd168b3a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1641058a0 con 0x7fd1640f51a0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd1637fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1641053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd1637fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1641053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:42780/0 (socket says 172.21.15.153:42780) 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd1637fe700 1 -- 172.21.15.153:0/2050397273 learned_addr learned my addr 172.21.15.153:0/2050397273 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd1637fe700 1 -- 172.21.15.153:0/2050397273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154005040 con 0x7fd1640f51a0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd1637fe700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1641053b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fd154013510 tx=0x7fd15400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd154009320 con 0x7fd1640f51a0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd154013f00 con 0x7fd1640f51a0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd154014dc0 con 0x7fd1640f51a0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164105aa0 con 0x7fd1640f51a0 2024-01-21T20:04:17.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.357+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd164105f30 con 0x7fd1640f51a0 2024-01-21T20:04:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.358+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd15401d3d0 con 0x7fd1640f51a0 2024-01-21T20:04:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.358+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd164045ad0 con 0x7fd1640f51a0 2024-01-21T20:04:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.358+0000 7fd14bfff700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd14c02ff60 0x7fd14c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.359+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd154041060 con 0x7fd1640f51a0 2024-01-21T20:04:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.359+0000 7fd162ffd700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd14c02ff60 0x7fd14c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:17.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.359+0000 7fd162ffd700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd14c02ff60 0x7fd14c032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd1580096a0 tx=0x7fd158006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:17.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.363+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1540095f0 con 0x7fd1640f51a0 2024-01-21T20:04:17.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.566+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd164106de0 con 0x7fd1640f51a0 2024-01-21T20:04:17.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.567+0000 7fd14bfff700 1 -- 172.21.15.153:0/2050397273 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd15401cb10 con 0x7fd1640f51a0 2024-01-21T20:04:17.558 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:17.559 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.568+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd14c02ff60 msgr2=0x7fd14c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd14c02ff60 0x7fd14c032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fd1580096a0 tx=0x7fd158006b40).stop 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 msgr2=0x7fd1641053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1641053b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fd154013510 tx=0x7fd15400b040).stop 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 shutdown_connections 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd14c02ff60 0x7fd14c032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 --2- 172.21.15.153:0/2050397273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1640f51a0 0x7fd1641053b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 >> 172.21.15.153:0/2050397273 conn(0x7fd1640f09a0 msgr2=0x7fd1640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 shutdown_connections 2024-01-21T20:04:17.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:17.569+0000 7fd168b3a700 1 -- 172.21.15.153:0/2050397273 wait complete. 2024-01-21T20:04:17.561 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:16.689217+0000 mgr.smithi119.rqacbm (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:18.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:17.568607+0000 mon.smithi119 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.153:0/2050397273' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:18.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:18.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:19.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:18.689583+0000 mgr.smithi119.rqacbm (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:20.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 -- 172.21.15.153:0/1258099132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 msgr2=0x7f0ad80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 --2- 172.21.15.153:0/1258099132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad80f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0acc009fb0 tx=0x7f0acc009b30).stop 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 -- 172.21.15.153:0/1258099132 shutdown_connections 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 --2- 172.21.15.153:0/1258099132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad80f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 -- 172.21.15.153:0/1258099132 >> 172.21.15.153:0/1258099132 conn(0x7f0ad80f0b70 msgr2=0x7f0ad80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 -- 172.21.15.153:0/1258099132 shutdown_connections 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 -- 172.21.15.153:0/1258099132 wait complete. 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.197+0000 7f0adf19b700 1 Processor -- start 2024-01-21T20:04:20.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adf19b700 1 -- start start 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adf19b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adf19b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8060220 con 0x7f0ad80f5410 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adcf37700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adcf37700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:42792/0 (socket says 172.21.15.153:42792) 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adcf37700 1 -- 172.21.15.153:0/1452757870 learned_addr learned my addr 172.21.15.153:0/1452757870 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adcf37700 1 -- 172.21.15.153:0/1452757870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc005040 con 0x7f0ad80f5410 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.198+0000 7f0adcf37700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad8109450 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0acc013510 tx=0x7f0acc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0acc009320 con 0x7f0ad80f5410 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad8060420 con 0x7f0ad80f5410 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0acc013f00 con 0x7f0ad80f5410 2024-01-21T20:04:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0acc014d90 con 0x7f0ad80f5410 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad8109b00 con 0x7f0ad80f5410 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0acc0263a0 con 0x7f0ad80f5410 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.199+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad8059100 con 0x7f0ad80f5410 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.200+0000 7f0ad57fa700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ac802ff60 0x7f0ac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.200+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0acc0412d0 con 0x7f0ad80f5410 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.200+0000 7f0ad7fff700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ac802ff60 0x7f0ac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:20.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.200+0000 7f0ad7fff700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ac802ff60 0x7f0ac8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0ac40096a0 tx=0x7f0ac4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:20.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.204+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0acc014600 con 0x7f0ad80f5410 2024-01-21T20:04:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.409+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ad8045ad0 con 0x7f0ad80f5410 2024-01-21T20:04:20.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.409+0000 7f0ad57fa700 1 -- 172.21.15.153:0/1452757870 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0acc014420 con 0x7f0ad80f5410 2024-01-21T20:04:20.400 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:20.400 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ac802ff60 msgr2=0x7f0ac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ac802ff60 0x7f0ac8032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0ac40096a0 tx=0x7f0ac4006b40).stop 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 msgr2=0x7f0ad8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad8109450 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f0acc013510 tx=0x7f0acc00b040).stop 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 shutdown_connections 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ac802ff60 0x7f0ac8032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 --2- 172.21.15.153:0/1452757870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ad80f5410 0x7f0ad8109450 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 >> 172.21.15.153:0/1452757870 conn(0x7f0ad80f0b70 msgr2=0x7f0ad80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 shutdown_connections 2024-01-21T20:04:20.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:20.411+0000 7f0adf19b700 1 -- 172.21.15.153:0/1452757870 wait complete. 2024-01-21T20:04:20.403 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:20.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:20.410678+0000 mon.smithi119 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.153:0/1452757870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:21.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:21.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:21.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:20.689953+0000 mgr.smithi119.rqacbm (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 -- 172.21.15.153:0/3775943090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 msgr2=0x7f7d4c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 --2- 172.21.15.153:0/3775943090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f9150 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f7d40009fb0 tx=0x7f7d40009b30).stop 2024-01-21T20:04:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 -- 172.21.15.153:0/3775943090 shutdown_connections 2024-01-21T20:04:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 --2- 172.21.15.153:0/3775943090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f9150 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 -- 172.21.15.153:0/3775943090 >> 172.21.15.153:0/3775943090 conn(0x7f7d4c0f0b70 msgr2=0x7f7d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 -- 172.21.15.153:0/3775943090 shutdown_connections 2024-01-21T20:04:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 -- 172.21.15.153:0/3775943090 wait complete. 2024-01-21T20:04:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.996+0000 7f7d52606700 1 Processor -- start 2024-01-21T20:04:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d52606700 1 -- start start 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d52606700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d52606700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c0f5050 con 0x7f7d4c060220 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d4bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d4bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:42818/0 (socket says 172.21.15.153:42818) 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d4bfff700 1 -- 172.21.15.153:0/185903662 learned_addr learned my addr 172.21.15.153:0/185903662 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d4bfff700 1 -- 172.21.15.153:0/185903662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d40005040 con 0x7f7d4c060220 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d4bfff700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f6bd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7d40013510 tx=0x7f7d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d40009320 con 0x7f7d4c060220 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d40013f00 con 0x7f7d4c060220 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c0f51f0 con 0x7f7d4c060220 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.997+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d40014dc0 con 0x7f7d4c060220 2024-01-21T20:04:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.998+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c0f5680 con 0x7f7d4c060220 2024-01-21T20:04:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.998+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d4001d3d0 con 0x7f7d4c060220 2024-01-21T20:04:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.998+0000 7f7d48ff9700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d3402ff60 0x7f7d34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.998+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d400410d0 con 0x7f7d4c060220 2024-01-21T20:04:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.998+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c059100 con 0x7f7d4c060220 2024-01-21T20:04:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.999+0000 7f7d4b7fe700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d3402ff60 0x7f7d34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:22.999+0000 7f7d4b7fe700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d3402ff60 0x7f7d34032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7d3c0096a0 tx=0x7f7d3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.004+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d40012900 con 0x7f7d4c060220 2024-01-21T20:04:23.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:22.690298+0000 mgr.smithi119.rqacbm (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:23.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.208+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d4c0f5d80 con 0x7f7d4c060220 2024-01-21T20:04:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.209+0000 7f7d48ff9700 1 -- 172.21.15.153:0/185903662 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d4002b020 con 0x7f7d4c060220 2024-01-21T20:04:23.200 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:23.200 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d3402ff60 msgr2=0x7f7d34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d3402ff60 0x7f7d34032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7d3c0096a0 tx=0x7f7d3c006b40).stop 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 msgr2=0x7f7d4c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f6bd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7d40013510 tx=0x7f7d4000b040).stop 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 shutdown_connections 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d3402ff60 0x7f7d34032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 --2- 172.21.15.153:0/185903662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4c060220 0x7f7d4c0f6bd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 >> 172.21.15.153:0/185903662 conn(0x7f7d4c0f0b70 msgr2=0x7f7d4c0f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 shutdown_connections 2024-01-21T20:04:23.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:23.211+0000 7f7d52606700 1 -- 172.21.15.153:0/185903662 wait complete. 2024-01-21T20:04:23.203 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:24.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:23.210612+0000 mon.smithi119 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.153:0/185903662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:24.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:24.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:25.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:24.690654+0000 mgr.smithi119.rqacbm (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/454717695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 msgr2=0x7f3b9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 --2- 172.21.15.153:0/454717695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c0f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f3b90009fb0 tx=0x7f3b90009b30).stop 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/454717695 shutdown_connections 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 --2- 172.21.15.153:0/454717695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c0f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/454717695 >> 172.21.15.153:0/454717695 conn(0x7f3b9c0f1c70 msgr2=0x7f3b9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/454717695 shutdown_connections 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/454717695 wait complete. 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 Processor -- start 2024-01-21T20:04:25.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.803+0000 7f3ba2ee4700 1 -- start start 2024-01-21T20:04:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba2ee4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba2ee4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c106c20 con 0x7f3b9c0f6510 2024-01-21T20:04:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba0c80700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba0c80700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43116/0 (socket says 172.21.15.153:43116) 2024-01-21T20:04:25.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba0c80700 1 -- 172.21.15.153:0/1321537088 learned_addr learned my addr 172.21.15.153:0/1321537088 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:25.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba0c80700 1 -- 172.21.15.153:0/1321537088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90005040 con 0x7f3b9c0f6510 2024-01-21T20:04:25.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3ba0c80700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c106730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f3b90013510 tx=0x7f3b9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:25.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.804+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b90009320 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.805+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b90013f00 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.805+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c106e20 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.805+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b90014d90 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.805+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c1072b0 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.805+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b900095f0 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.806+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c045ad0 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.806+0000 7f3b997fa700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3b8c02ff60 0x7f3b8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.806+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b90041000 con 0x7f3b9c0f6510 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.806+0000 7f3b9bfff700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3b8c02ff60 0x7f3b8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:25.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.806+0000 7f3b9bfff700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3b8c02ff60 0x7f3b8c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3b880096a0 tx=0x7f3b88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:25.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:25.811+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b90014420 con 0x7f3b9c0f6510 2024-01-21T20:04:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.017+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b9c0fbf60 con 0x7f3b9c0f6510 2024-01-21T20:04:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3b997fa700 1 -- 172.21.15.153:0/1321537088 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3b9001d7f0 con 0x7f3b9c0f6510 2024-01-21T20:04:26.009 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:26.009 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3b8c02ff60 msgr2=0x7f3b8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3b8c02ff60 0x7f3b8c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3b880096a0 tx=0x7f3b88006b40).stop 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 msgr2=0x7f3b9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c106730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f3b90013510 tx=0x7f3b9000b040).stop 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 shutdown_connections 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3b8c02ff60 0x7f3b8c032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 --2- 172.21.15.153:0/1321537088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b9c0f6510 0x7f3b9c106730 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.019+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 >> 172.21.15.153:0/1321537088 conn(0x7f3b9c0f1c70 msgr2=0x7f3b9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.020+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 shutdown_connections 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:26.020+0000 7f3ba2ee4700 1 -- 172.21.15.153:0/1321537088 wait complete. 2024-01-21T20:04:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:26.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:26 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:26.019104+0000 mon.smithi119 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.153:0/1321537088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:27.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:27.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:26.690941+0000 mgr.smithi119.rqacbm (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 -- 172.21.15.153:0/1198307847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 msgr2=0x7fac540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 --2- 172.21.15.153:0/1198307847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac540f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fac3c009fb0 tx=0x7fac3c009b30).stop 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 -- 172.21.15.153:0/1198307847 shutdown_connections 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 --2- 172.21.15.153:0/1198307847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac540f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 -- 172.21.15.153:0/1198307847 >> 172.21.15.153:0/1198307847 conn(0x7fac540f0b70 msgr2=0x7fac540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 -- 172.21.15.153:0/1198307847 shutdown_connections 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.596+0000 7fac589b6700 1 -- 172.21.15.153:0/1198307847 wait complete. 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac589b6700 1 Processor -- start 2024-01-21T20:04:28.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac589b6700 1 -- start start 2024-01-21T20:04:28.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac589b6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac54106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:28.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac589b6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54106c10 con 0x7fac540f5410 2024-01-21T20:04:28.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac5259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac54106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:28.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac5259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac54106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43128/0 (socket says 172.21.15.153:43128) 2024-01-21T20:04:28.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac5259c700 1 -- 172.21.15.153:0/3338168021 learned_addr learned my addr 172.21.15.153:0/3338168021 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:28.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac5259c700 1 -- 172.21.15.153:0/3338168021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c005040 con 0x7fac540f5410 2024-01-21T20:04:28.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac5259c700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac54106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fac3c014000 tx=0x7fac3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:28.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.597+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3c0095f0 con 0x7fac540f5410 2024-01-21T20:04:28.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac54106e10 con 0x7fac540f5410 2024-01-21T20:04:28.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac3c007e70 con 0x7fac540f5410 2024-01-21T20:04:28.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3c014ea0 con 0x7fac540f5410 2024-01-21T20:04:28.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac541072a0 con 0x7fac540f5410 2024-01-21T20:04:28.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac3c025070 con 0x7fac540f5410 2024-01-21T20:04:28.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac54045ad0 con 0x7fac540f5410 2024-01-21T20:04:28.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.598+0000 7fac4affd700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fac4002ff60 0x7fac40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:28.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.599+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac3c040970 con 0x7fac540f5410 2024-01-21T20:04:28.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.599+0000 7fac51d9b700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fac4002ff60 0x7fac40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:28.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.599+0000 7fac51d9b700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fac4002ff60 0x7fac40032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fac440096a0 tx=0x7fac44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:28.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.603+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac3c0079c0 con 0x7fac540f5410 2024-01-21T20:04:28.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.809+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac54108150 con 0x7fac540f5410 2024-01-21T20:04:28.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.810+0000 7fac4affd700 1 -- 172.21.15.153:0/3338168021 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fac3c00f300 con 0x7fac540f5410 2024-01-21T20:04:28.801 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:28.801 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:28.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fac4002ff60 msgr2=0x7fac40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:28.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fac4002ff60 0x7fac40032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fac440096a0 tx=0x7fac44006b40).stop 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 msgr2=0x7fac54106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac54106720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fac3c014000 tx=0x7fac3c00b040).stop 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 shutdown_connections 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fac4002ff60 0x7fac40032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 --2- 172.21.15.153:0/3338168021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac540f5410 0x7fac54106720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 >> 172.21.15.153:0/3338168021 conn(0x7fac540f0b70 msgr2=0x7fac5405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 shutdown_connections 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:28.812+0000 7fac589b6700 1 -- 172.21.15.153:0/3338168021 wait complete. 2024-01-21T20:04:28.803 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:29.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:28.691204+0000 mgr.smithi119.rqacbm (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:30.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:28.810764+0000 mon.smithi119 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.153:0/3338168021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:30.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:30.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:31.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:30.691557+0000 mgr.smithi119.rqacbm (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 -- 172.21.15.153:0/1913326756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 msgr2=0x7fbea40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 --2- 172.21.15.153:0/1913326756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea40f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fbe94009fb0 tx=0x7fbe94009b30).stop 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 -- 172.21.15.153:0/1913326756 shutdown_connections 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 --2- 172.21.15.153:0/1913326756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea40f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 -- 172.21.15.153:0/1913326756 >> 172.21.15.153:0/1913326756 conn(0x7fbea40f1c70 msgr2=0x7fbea40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 -- 172.21.15.153:0/1913326756 shutdown_connections 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.385+0000 7fbeabf3e700 1 -- 172.21.15.153:0/1913326756 wait complete. 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbeabf3e700 1 Processor -- start 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbeabf3e700 1 -- start start 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbeabf3e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbeabf3e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea4106c20 con 0x7fbea40f6510 2024-01-21T20:04:31.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbea9cda700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbea9cda700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43144/0 (socket says 172.21.15.153:43144) 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbea9cda700 1 -- 172.21.15.153:0/2429469354 learned_addr learned my addr 172.21.15.153:0/2429469354 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbea9cda700 1 -- 172.21.15.153:0/2429469354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe94005040 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.386+0000 7fbea9cda700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea4106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fbe94013510 tx=0x7fbe9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe94009320 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea4106e20 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe94013f00 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe94014d90 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea41072b0 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe940095f0 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.387+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea4045ad0 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.388+0000 7fbe9a7fc700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbe9002ff60 0x7fbe90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.388+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe94040fc0 con 0x7fbea40f6510 2024-01-21T20:04:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.388+0000 7fbea94d9700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbe9002ff60 0x7fbe90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:31.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.388+0000 7fbea94d9700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbe9002ff60 0x7fbe90032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:31.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.392+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe94014420 con 0x7fbea40f6510 2024-01-21T20:04:31.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.598+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbea40fbf60 con 0x7fbea40f6510 2024-01-21T20:04:31.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.599+0000 7fbe9a7fc700 1 -- 172.21.15.153:0/2429469354 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbe94010230 con 0x7fbea40f6510 2024-01-21T20:04:31.589 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:31.589 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbe9002ff60 msgr2=0x7fbe90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbe9002ff60 0x7fbe90032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fbea00096a0 tx=0x7fbea0006b40).stop 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 msgr2=0x7fbea4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea4106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fbe94013510 tx=0x7fbe9400b040).stop 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 shutdown_connections 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbe9002ff60 0x7fbe90032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 --2- 172.21.15.153:0/2429469354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbea40f6510 0x7fbea4106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:31.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 >> 172.21.15.153:0/2429469354 conn(0x7fbea40f1c70 msgr2=0x7fbea40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:31.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 shutdown_connections 2024-01-21T20:04:31.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:31.601+0000 7fbeabf3e700 1 -- 172.21.15.153:0/2429469354 wait complete. 2024-01-21T20:04:31.592 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:32.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:31.599431+0000 mon.smithi119 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.153:0/2429469354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:32.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:32.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:33.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:32.691834+0000 mgr.smithi119.rqacbm (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 -- 172.21.15.153:0/1511719576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 msgr2=0x7f82480fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:34.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 --2- 172.21.15.153:0/1511719576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82480fb570 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f823c009fb0 tx=0x7f823c009b30).stop 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 -- 172.21.15.153:0/1511719576 shutdown_connections 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 --2- 172.21.15.153:0/1511719576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82480fb570 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 -- 172.21.15.153:0/1511719576 >> 172.21.15.153:0/1511719576 conn(0x7f82480f0b70 msgr2=0x7f82480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 -- 172.21.15.153:0/1511719576 shutdown_connections 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 -- 172.21.15.153:0/1511719576 wait complete. 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 Processor -- start 2024-01-21T20:04:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.169+0000 7f824eb6c700 1 -- start start 2024-01-21T20:04:34.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824eb6c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82481088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:34.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824eb6c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8248108dc0 con 0x7f8248060200 2024-01-21T20:04:34.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824c908700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82481088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:34.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824c908700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82481088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47238/0 (socket says 172.21.15.153:47238) 2024-01-21T20:04:34.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824c908700 1 -- 172.21.15.153:0/1007491595 learned_addr learned my addr 172.21.15.153:0/1007491595 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:34.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824c908700 1 -- 172.21.15.153:0/1007491595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c005040 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f824c908700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82481088d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f823c013510 tx=0x7f823c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.170+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f823c009320 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8248108fc0 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f823c013f00 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f823c014dc0 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8248109450 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f823c0095f0 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8248045ad0 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.171+0000 7f82457fa700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f823802ff60 0x7f8238032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.172+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f823c041100 con 0x7f8248060200 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.172+0000 7f8247fff700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f823802ff60 0x7f8238032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:34.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.172+0000 7f8247fff700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f823802ff60 0x7f8238032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f82340096a0 tx=0x7f8234006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:34.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.176+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f823c0268c0 con 0x7f8248060200 2024-01-21T20:04:34.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.386+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f824810a300 con 0x7f8248060200 2024-01-21T20:04:34.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.387+0000 7f82457fa700 1 -- 172.21.15.153:0/1007491595 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f823c010230 con 0x7f8248060200 2024-01-21T20:04:34.377 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:34.377 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f823802ff60 msgr2=0x7f8238032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f823802ff60 0x7f8238032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f82340096a0 tx=0x7f8234006b40).stop 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 msgr2=0x7f82481088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82481088d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f823c013510 tx=0x7f823c00b040).stop 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 shutdown_connections 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f823802ff60 0x7f8238032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 --2- 172.21.15.153:0/1007491595 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8248060200 0x7f82481088d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 >> 172.21.15.153:0/1007491595 conn(0x7f82480f0b70 msgr2=0x7f82480f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 shutdown_connections 2024-01-21T20:04:34.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:34.389+0000 7f824eb6c700 1 -- 172.21.15.153:0/1007491595 wait complete. 2024-01-21T20:04:34.380 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:34.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:34.387671+0000 mon.smithi119 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.153:0/1007491595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:35.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:35.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:35.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:34.692105+0000 mgr.smithi119.rqacbm (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 -- 172.21.15.153:0/1261105860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 msgr2=0x7f4a7c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 --2- 172.21.15.153:0/1261105860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c0f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f4a6c009fb0 tx=0x7f4a6c009b30).stop 2024-01-21T20:04:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 -- 172.21.15.153:0/1261105860 shutdown_connections 2024-01-21T20:04:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 --2- 172.21.15.153:0/1261105860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c0f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 -- 172.21.15.153:0/1261105860 >> 172.21.15.153:0/1261105860 conn(0x7f4a7c0f1c70 msgr2=0x7f4a7c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:36.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 -- 172.21.15.153:0/1261105860 shutdown_connections 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.986+0000 7f4a845f2700 1 -- 172.21.15.153:0/1261105860 wait complete. 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a845f2700 1 Processor -- start 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a845f2700 1 -- start start 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a845f2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a845f2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7c106c20 con 0x7f4a7c0f6510 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a8238e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a8238e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47258/0 (socket says 172.21.15.153:47258) 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.987+0000 7f4a8238e700 1 -- 172.21.15.153:0/268412437 learned_addr learned my addr 172.21.15.153:0/268412437 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a8238e700 1 -- 172.21.15.153:0/268412437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c005040 con 0x7f4a7c0f6510 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a8238e700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f4a6c013510 tx=0x7f4a6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:36.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c009320 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a7c106e20 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a6c013f00 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a6c014d90 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.988+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7c1072b0 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.989+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a6c0095f0 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.989+0000 7f4a72ffd700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4a6802ff60 0x7f4a68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.989+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a6c0410c0 con 0x7f4a7c0f6510 2024-01-21T20:04:36.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.989+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7c045ad0 con 0x7f4a7c0f6510 2024-01-21T20:04:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.989+0000 7f4a81b8d700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4a6802ff60 0x7f4a68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:36.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.990+0000 7f4a81b8d700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4a6802ff60 0x7f4a68032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4a780096a0 tx=0x7f4a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:36.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:36.994+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a6c014420 con 0x7f4a7c0f6510 2024-01-21T20:04:37.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.197+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a7c108070 con 0x7f4a7c0f6510 2024-01-21T20:04:37.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.198+0000 7f4a72ffd700 1 -- 172.21.15.153:0/268412437 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a6c02b020 con 0x7f4a7c0f6510 2024-01-21T20:04:37.188 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:37.189 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4a6802ff60 msgr2=0x7f4a68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4a6802ff60 0x7f4a68032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f4a780096a0 tx=0x7f4a78006b40).stop 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 msgr2=0x7f4a7c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f4a6c013510 tx=0x7f4a6c00b040).stop 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 shutdown_connections 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4a6802ff60 0x7f4a68032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:37.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 --2- 172.21.15.153:0/268412437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a7c0f6510 0x7f4a7c106730 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:37.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 >> 172.21.15.153:0/268412437 conn(0x7f4a7c0f1c70 msgr2=0x7f4a7c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:37.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 shutdown_connections 2024-01-21T20:04:37.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:37.199+0000 7f4a845f2700 1 -- 172.21.15.153:0/268412437 wait complete. 2024-01-21T20:04:37.191 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:36.692399+0000 mgr.smithi119.rqacbm (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:38.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:38.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:38.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:37.198237+0000 mon.smithi119 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.153:0/268412437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:39.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:38.692692+0000 mgr.smithi119.rqacbm (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 -- 172.21.15.153:0/769417498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 msgr2=0x7f4dec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 --2- 172.21.15.153:0/769417498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f4ddc009fb0 tx=0x7f4ddc009b30).stop 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 -- 172.21.15.153:0/769417498 shutdown_connections 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 --2- 172.21.15.153:0/769417498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 -- 172.21.15.153:0/769417498 >> 172.21.15.153:0/769417498 conn(0x7f4dec0f0b70 msgr2=0x7f4dec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 -- 172.21.15.153:0/769417498 shutdown_connections 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 -- 172.21.15.153:0/769417498 wait complete. 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.791+0000 7f4df4032700 1 Processor -- start 2024-01-21T20:04:39.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df4032700 1 -- start start 2024-01-21T20:04:39.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df4032700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:39.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df4032700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec106c10 con 0x7f4dec0f5410 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df1dce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df1dce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47278/0 (socket says 172.21.15.153:47278) 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df1dce700 1 -- 172.21.15.153:0/3556054993 learned_addr learned my addr 172.21.15.153:0/3556054993 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.792+0000 7f4df1dce700 1 -- 172.21.15.153:0/3556054993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc005040 con 0x7f4dec0f5410 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.793+0000 7f4df1dce700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec106720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4ddc013510 tx=0x7f4ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.793+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ddc009320 con 0x7f4dec0f5410 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.793+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dec106e10 con 0x7f4dec0f5410 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.793+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ddc013f00 con 0x7f4dec0f5410 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.793+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ddc014d90 con 0x7f4dec0f5410 2024-01-21T20:04:39.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.793+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dec1072a0 con 0x7f4dec0f5410 2024-01-21T20:04:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.794+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ddc014420 con 0x7f4dec0f5410 2024-01-21T20:04:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.794+0000 7f4de27fc700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4dd802ff60 0x7f4dd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.794+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ddc040fb0 con 0x7f4dec0f5410 2024-01-21T20:04:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.794+0000 7f4df15cd700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4dd802ff60 0x7f4dd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.795+0000 7f4df15cd700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4dd802ff60 0x7f4dd8032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4de80096a0 tx=0x7f4de8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:39.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.795+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dec059100 con 0x7f4dec0f5410 2024-01-21T20:04:39.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:39.800+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ddc0095f0 con 0x7f4dec0f5410 2024-01-21T20:04:39.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.006+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4dec0fae60 con 0x7f4dec0f5410 2024-01-21T20:04:39.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.006+0000 7f4de27fc700 1 -- 172.21.15.153:0/3556054993 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ddc01db70 con 0x7f4dec0f5410 2024-01-21T20:04:39.998 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:39.998 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:39.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4dd802ff60 msgr2=0x7f4dd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:39.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4dd802ff60 0x7f4dd8032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4de80096a0 tx=0x7f4de8006b40).stop 2024-01-21T20:04:39.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 msgr2=0x7f4dec106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:39.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec106720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4ddc013510 tx=0x7f4ddc00b040).stop 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 shutdown_connections 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4dd802ff60 0x7f4dd8032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 --2- 172.21.15.153:0/3556054993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dec0f5410 0x7f4dec106720 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 >> 172.21.15.153:0/3556054993 conn(0x7f4dec0f0b70 msgr2=0x7f4dec05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 shutdown_connections 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:40.008+0000 7f4df4032700 1 -- 172.21.15.153:0/3556054993 wait complete. 2024-01-21T20:04:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:40.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:40.007947+0000 mon.smithi119 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.153:0/3556054993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:41.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:41.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:41.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:40.692987+0000 mgr.smithi119.rqacbm (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:42.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 -- 172.21.15.153:0/3962867563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380f5410 msgr2=0x7f66380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 --2- 172.21.15.153:0/3962867563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380f5410 0x7f66380f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f662c009fb0 tx=0x7f662c009b30).stop 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 -- 172.21.15.153:0/3962867563 shutdown_connections 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 --2- 172.21.15.153:0/3962867563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380f5410 0x7f66380f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 -- 172.21.15.153:0/3962867563 >> 172.21.15.153:0/3962867563 conn(0x7f66380f0b70 msgr2=0x7f66380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 -- 172.21.15.153:0/3962867563 shutdown_connections 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 -- 172.21.15.153:0/3962867563 wait complete. 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.548+0000 7f663d06a700 1 Processor -- start 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f663d06a700 1 -- start start 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f663d06a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 0x7f66380fb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f663d06a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66380fd800 con 0x7f66380fd3e0 2024-01-21T20:04:42.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f6636d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 0x7f66380fb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f6636d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 0x7f66380fb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47308/0 (socket says 172.21.15.153:47308) 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f6636d9d700 1 -- 172.21.15.153:0/3724681212 learned_addr learned my addr 172.21.15.153:0/3724681212 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f6636d9d700 1 -- 172.21.15.153:0/3724681212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c005040 con 0x7f66380fd3e0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f6636d9d700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 0x7f66380fb8a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f662c009f80 tx=0x7f662c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f662c0136c0 con 0x7f66380fd3e0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f662c013e70 con 0x7f66380fd3e0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66380fbd90 con 0x7f66380fd3e0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f662c014420 con 0x7f66380fd3e0 2024-01-21T20:04:42.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.549+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66380fc280 con 0x7f66380fd3e0 2024-01-21T20:04:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.550+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f662c025060 con 0x7f66380fd3e0 2024-01-21T20:04:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.550+0000 7f661f7fe700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f662002ff60 0x7f6620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.550+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f662c040f30 con 0x7f66380fd3e0 2024-01-21T20:04:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.550+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6638059100 con 0x7f66380fd3e0 2024-01-21T20:04:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.551+0000 7f663659c700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f662002ff60 0x7f6620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:42.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.551+0000 7f663659c700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f662002ff60 0x7f6620032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f66280096a0 tx=0x7f6628006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:42.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.555+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f662c028440 con 0x7f66380fd3e0 2024-01-21T20:04:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.763+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6638045ad0 con 0x7f66380fd3e0 2024-01-21T20:04:42.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.764+0000 7f661f7fe700 1 -- 172.21.15.153:0/3724681212 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f662c004490 con 0x7f66380fd3e0 2024-01-21T20:04:42.755 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:42.755 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f662002ff60 msgr2=0x7f6620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f662002ff60 0x7f6620032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f66280096a0 tx=0x7f6628006b40).stop 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 msgr2=0x7f66380fb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 0x7f66380fb8a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f662c009f80 tx=0x7f662c009b30).stop 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 shutdown_connections 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f662002ff60 0x7f6620032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 --2- 172.21.15.153:0/3724681212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66380fd3e0 0x7f66380fb8a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:42.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 >> 172.21.15.153:0/3724681212 conn(0x7f66380f0b70 msgr2=0x7f663805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:42.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 shutdown_connections 2024-01-21T20:04:42.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:42.766+0000 7f663d06a700 1 -- 172.21.15.153:0/3724681212 wait complete. 2024-01-21T20:04:42.758 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:43.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:42.693278+0000 mgr.smithi119.rqacbm (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:43.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:42.765499+0000 mon.smithi119 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.153:0/3724681212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:44.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:44.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:45.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:44.693612+0000 mgr.smithi119.rqacbm (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3964721277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 msgr2=0x7f4c7c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 --2- 172.21.15.153:0/3964721277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c0f8f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f4c64009fb0 tx=0x7f4c64009b30).stop 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3964721277 shutdown_connections 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 --2- 172.21.15.153:0/3964721277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c0f8f20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3964721277 >> 172.21.15.153:0/3964721277 conn(0x7f4c7c0f09a0 msgr2=0x7f4c7c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3964721277 shutdown_connections 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3964721277 wait complete. 2024-01-21T20:04:45.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.340+0000 7f4c80ba0700 1 Processor -- start 2024-01-21T20:04:45.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c80ba0700 1 -- start start 2024-01-21T20:04:45.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c80ba0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:45.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c80ba0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c1069f0 con 0x7f4c7c0601e0 2024-01-21T20:04:45.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c7b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:45.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c7b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:48250/0 (socket says 172.21.15.153:48250) 2024-01-21T20:04:45.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c7b7fe700 1 -- 172.21.15.153:0/3142479142 learned_addr learned my addr 172.21.15.153:0/3142479142 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:45.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c7b7fe700 1 -- 172.21.15.153:0/3142479142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c64005040 con 0x7f4c7c0601e0 2024-01-21T20:04:45.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.341+0000 7f4c7b7fe700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c106500 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f4c64013510 tx=0x7f4c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:45.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.342+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c64009320 con 0x7f4c7c0601e0 2024-01-21T20:04:45.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.342+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7c106bf0 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.342+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c64013f00 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.342+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c64014d90 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.342+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7c107080 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.342+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c640263a0 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.343+0000 7f4c73fff700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4c6802ff60 0x7f4c68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.343+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c64040fe0 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.343+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7c045ad0 con 0x7f4c7c0601e0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.343+0000 7f4c7affd700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4c6802ff60 0x7f4c68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:45.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.343+0000 7f4c7affd700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4c6802ff60 0x7f4c68032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4c6c0096a0 tx=0x7f4c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:45.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.347+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c64012920 con 0x7f4c7c0601e0 2024-01-21T20:04:45.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.553+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c7c107da0 con 0x7f4c7c0601e0 2024-01-21T20:04:45.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.554+0000 7f4c73fff700 1 -- 172.21.15.153:0/3142479142 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4c6402b020 con 0x7f4c7c0601e0 2024-01-21T20:04:45.545 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:45.545 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:45.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.555+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4c6802ff60 msgr2=0x7f4c68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.555+0000 7f4c80ba0700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4c6802ff60 0x7f4c68032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f4c6c0096a0 tx=0x7f4c6c006b40).stop 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.555+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 msgr2=0x7f4c7c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.555+0000 7f4c80ba0700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c106500 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f4c64013510 tx=0x7f4c6400b040).stop 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.556+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 shutdown_connections 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.556+0000 7f4c80ba0700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4c6802ff60 0x7f4c68032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.556+0000 7f4c80ba0700 1 --2- 172.21.15.153:0/3142479142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7c0601e0 0x7f4c7c106500 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.556+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 >> 172.21.15.153:0/3142479142 conn(0x7f4c7c0f09a0 msgr2=0x7f4c7c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.556+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 shutdown_connections 2024-01-21T20:04:45.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:45.556+0000 7f4c80ba0700 1 -- 172.21.15.153:0/3142479142 wait complete. 2024-01-21T20:04:45.548 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:46.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:45 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:45.555359+0000 mon.smithi119 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.153:0/3142479142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:46.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:46.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:46.693978+0000 mgr.smithi119.rqacbm (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 -- 172.21.15.153:0/2163713833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 msgr2=0x7f6f6c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 --2- 172.21.15.153:0/2163713833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f90f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f6f68009fb0 tx=0x7f6f68009b30).stop 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 -- 172.21.15.153:0/2163713833 shutdown_connections 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 --2- 172.21.15.153:0/2163713833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f90f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 -- 172.21.15.153:0/2163713833 >> 172.21.15.153:0/2163713833 conn(0x7f6f6c0f0b70 msgr2=0x7f6f6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 -- 172.21.15.153:0/2163713833 shutdown_connections 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 -- 172.21.15.153:0/2163713833 wait complete. 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 Processor -- start 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 -- start start 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.148+0000 7f6f73bfa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f73bfa700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c0f4ff0 con 0x7f6f6c060280 2024-01-21T20:04:48.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f71996700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:48.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f71996700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:48266/0 (socket says 172.21.15.153:48266) 2024-01-21T20:04:48.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f71996700 1 -- 172.21.15.153:0/3409636001 learned_addr learned my addr 172.21.15.153:0/3409636001 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:48.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f71996700 1 -- 172.21.15.153:0/3409636001 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f68005040 con 0x7f6f6c060280 2024-01-21T20:04:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f71996700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6f68013510 tx=0x7f6f6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f68009320 con 0x7f6f6c060280 2024-01-21T20:04:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c0f5190 con 0x7f6f6c060280 2024-01-21T20:04:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f68013f00 con 0x7f6f6c060280 2024-01-21T20:04:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f68014dc0 con 0x7f6f6c060280 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.149+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c0f5620 con 0x7f6f6c060280 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.150+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f68014420 con 0x7f6f6c060280 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.150+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c059100 con 0x7f6f6c060280 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.150+0000 7f6f627fc700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f5802ff60 0x7f6f58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.150+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f68041030 con 0x7f6f6c060280 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.151+0000 7f6f71195700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f5802ff60 0x7f6f58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.151+0000 7f6f71195700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f5802ff60 0x7f6f58032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6f5c0096a0 tx=0x7f6f5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.155+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f680095f0 con 0x7f6f6c060280 2024-01-21T20:04:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.358+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f6c0fbf50 con 0x7f6f6c060280 2024-01-21T20:04:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.359+0000 7f6f627fc700 1 -- 172.21.15.153:0/3409636001 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f6801cb10 con 0x7f6f6c060280 2024-01-21T20:04:48.350 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:48.350 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f5802ff60 msgr2=0x7f6f58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f5802ff60 0x7f6f58032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6f5c0096a0 tx=0x7f6f5c006b40).stop 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 msgr2=0x7f6f6c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6f68013510 tx=0x7f6f6800b040).stop 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 shutdown_connections 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f5802ff60 0x7f6f58032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 --2- 172.21.15.153:0/3409636001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f6c060280 0x7f6f6c0f6b70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 >> 172.21.15.153:0/3409636001 conn(0x7f6f6c0f0b70 msgr2=0x7f6f6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 shutdown_connections 2024-01-21T20:04:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:48.361+0000 7f6f73bfa700 1 -- 172.21.15.153:0/3409636001 wait complete. 2024-01-21T20:04:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:48.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:48.360625+0000 mon.smithi119 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.153:0/3409636001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:49.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:49.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:49.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:48.694381+0000 mgr.smithi119.rqacbm (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:50.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 -- 172.21.15.153:0/1779671872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 msgr2=0x7fcb7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:50.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 --2- 172.21.15.153:0/1779671872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c0f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fcb74009fb0 tx=0x7fcb74009b30).stop 2024-01-21T20:04:50.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 -- 172.21.15.153:0/1779671872 shutdown_connections 2024-01-21T20:04:50.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 --2- 172.21.15.153:0/1779671872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c0f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:50.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 -- 172.21.15.153:0/1779671872 >> 172.21.15.153:0/1779671872 conn(0x7fcb7c0f0b70 msgr2=0x7fcb7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:50.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 -- 172.21.15.153:0/1779671872 shutdown_connections 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.925+0000 7fcb837e1700 1 -- 172.21.15.153:0/1779671872 wait complete. 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb837e1700 1 Processor -- start 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb837e1700 1 -- start start 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb837e1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb837e1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c106c00 con 0x7fcb7c0f5410 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb8157d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:50.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb8157d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:48296/0 (socket says 172.21.15.153:48296) 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.926+0000 7fcb8157d700 1 -- 172.21.15.153:0/1477350018 learned_addr learned my addr 172.21.15.153:0/1477350018 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb8157d700 1 -- 172.21.15.153:0/1477350018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb74005040 con 0x7fcb7c0f5410 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb8157d700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c106710 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fcb74013510 tx=0x7fcb7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb74009320 con 0x7fcb7c0f5410 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb74013f00 con 0x7fcb7c0f5410 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb74014d90 con 0x7fcb7c0f5410 2024-01-21T20:04:50.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c106e00 con 0x7fcb7c0f5410 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.927+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c107290 con 0x7fcb7c0f5410 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.928+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb740095f0 con 0x7fcb7c0f5410 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.928+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c045ad0 con 0x7fcb7c0f5410 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.928+0000 7fcb79ffb700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcb6402ff60 0x7fcb64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.928+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb740410c0 con 0x7fcb7c0f5410 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.928+0000 7fcb80d7c700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcb6402ff60 0x7fcb64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:50.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.929+0000 7fcb80d7c700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcb6402ff60 0x7fcb64032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcb700096a0 tx=0x7fcb70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:50.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:50.933+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb7401d7f0 con 0x7fcb7c0f5410 2024-01-21T20:04:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:50.694723+0000 mgr.smithi119.rqacbm (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:51.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.140+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb7c0fae60 con 0x7fcb7c0f5410 2024-01-21T20:04:51.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.140+0000 7fcb79ffb700 1 -- 172.21.15.153:0/1477350018 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb74010230 con 0x7fcb7c0f5410 2024-01-21T20:04:51.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:51.132 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcb6402ff60 msgr2=0x7fcb64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcb6402ff60 0x7fcb64032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcb700096a0 tx=0x7fcb70006b40).stop 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 msgr2=0x7fcb7c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c106710 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fcb74013510 tx=0x7fcb7400b040).stop 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 shutdown_connections 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcb6402ff60 0x7fcb64032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:51.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 --2- 172.21.15.153:0/1477350018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb7c0f5410 0x7fcb7c106710 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:51.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 >> 172.21.15.153:0/1477350018 conn(0x7fcb7c0f0b70 msgr2=0x7fcb7c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:51.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 shutdown_connections 2024-01-21T20:04:51.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:51.142+0000 7fcb837e1700 1 -- 172.21.15.153:0/1477350018 wait complete. 2024-01-21T20:04:51.134 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:52.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:51.141812+0000 mon.smithi119 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.153:0/1477350018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:52.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:52.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:53.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:52.695037+0000 mgr.smithi119.rqacbm (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:53.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.696+0000 7f94e1f84700 1 -- 172.21.15.153:0/860728673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 msgr2=0x7f94dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:53.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 --2- 172.21.15.153:0/860728673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc0f55c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f94d0009fb0 tx=0x7f94d0009b30).stop 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 -- 172.21.15.153:0/860728673 shutdown_connections 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 --2- 172.21.15.153:0/860728673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc0f55c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 -- 172.21.15.153:0/860728673 >> 172.21.15.153:0/860728673 conn(0x7f94dc0f09a0 msgr2=0x7f94dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 -- 172.21.15.153:0/860728673 shutdown_connections 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 -- 172.21.15.153:0/860728673 wait complete. 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 Processor -- start 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.697+0000 7f94e1f84700 1 -- start start 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e1f84700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e1f84700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc060240 con 0x7f94dc0f51a0 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e0f82700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e0f82700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:55840/0 (socket says 172.21.15.153:55840) 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e0f82700 1 -- 172.21.15.153:0/3901786764 learned_addr learned my addr 172.21.15.153:0/3901786764 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e0f82700 1 -- 172.21.15.153:0/3901786764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0005040 con 0x7f94dc0f51a0 2024-01-21T20:04:53.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e0f82700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc1091b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f94d0009830 tx=0x7f94d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:53.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94d0009320 con 0x7f94dc0f51a0 2024-01-21T20:04:53.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94dc060440 con 0x7f94dc0f51a0 2024-01-21T20:04:53.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94d00136c0 con 0x7f94dc0f51a0 2024-01-21T20:04:53.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94d001ee90 con 0x7f94dc0f51a0 2024-01-21T20:04:53.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.698+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94dc109860 con 0x7f94dc0f51a0 2024-01-21T20:04:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.699+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94d001e420 con 0x7f94dc0f51a0 2024-01-21T20:04:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.699+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94dc045ad0 con 0x7f94dc0f51a0 2024-01-21T20:04:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.699+0000 7f94d97fa700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f94cc02ff10 0x7f94cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.699+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94d0040f10 con 0x7f94dc0f51a0 2024-01-21T20:04:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.700+0000 7f94dbfff700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f94cc02ff10 0x7f94cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:53.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.700+0000 7f94dbfff700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f94cc02ff10 0x7f94cc0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f94c80096a0 tx=0x7f94c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:53.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.704+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94d00095f0 con 0x7f94dc0f51a0 2024-01-21T20:04:53.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.914+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94dc10a080 con 0x7f94dc0f51a0 2024-01-21T20:04:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.915+0000 7f94d97fa700 1 -- 172.21.15.153:0/3901786764 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f94d002c2d0 con 0x7f94dc0f51a0 2024-01-21T20:04:53.906 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:53.907 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f94cc02ff10 msgr2=0x7f94cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f94cc02ff10 0x7f94cc0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f94c80096a0 tx=0x7f94c8006b40).stop 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 msgr2=0x7f94dc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc1091b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f94d0009830 tx=0x7f94d000b040).stop 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 shutdown_connections 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f94cc02ff10 0x7f94cc0323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:53.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 --2- 172.21.15.153:0/3901786764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94dc0f51a0 0x7f94dc1091b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 >> 172.21.15.153:0/3901786764 conn(0x7f94dc0f09a0 msgr2=0x7f94dc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 shutdown_connections 2024-01-21T20:04:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:53.917+0000 7f94e1f84700 1 -- 172.21.15.153:0/3901786764 wait complete. 2024-01-21T20:04:53.909 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:54.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:53.916717+0000 mon.smithi119 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.153:0/3901786764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:55.305 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:55.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:55.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:54.695453+0000 mgr.smithi119.rqacbm (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 -- 172.21.15.153:0/680431156 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 msgr2=0x7ff9f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 --2- 172.21.15.153:0/680431156 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f80f90f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7ff9e8009fb0 tx=0x7ff9e8009b30).stop 2024-01-21T20:04:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 -- 172.21.15.153:0/680431156 shutdown_connections 2024-01-21T20:04:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 --2- 172.21.15.153:0/680431156 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f80f90f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:56.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 -- 172.21.15.153:0/680431156 >> 172.21.15.153:0/680431156 conn(0x7ff9f80f0b70 msgr2=0x7ff9f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 -- 172.21.15.153:0/680431156 shutdown_connections 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.550+0000 7ff9fdea7700 1 -- 172.21.15.153:0/680431156 wait complete. 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9fdea7700 1 Processor -- start 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9fdea7700 1 -- start start 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9fdea7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9fdea7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8106bd0 con 0x7ff9f8060280 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9f77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9f77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:55856/0 (socket says 172.21.15.153:55856) 2024-01-21T20:04:56.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.551+0000 7ff9f77fe700 1 -- 172.21.15.153:0/2850327099 learned_addr learned my addr 172.21.15.153:0/2850327099 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9f77fe700 1 -- 172.21.15.153:0/2850327099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8005040 con 0x7ff9f8060280 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9f77fe700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f81066e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ff9e8013510 tx=0x7ff9e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e8009320 con 0x7ff9f8060280 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9e8013f00 con 0x7ff9f8060280 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f8106dd0 con 0x7ff9f8060280 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e8014dc0 con 0x7ff9f8060280 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.552+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8107260 con 0x7ff9f8060280 2024-01-21T20:04:56.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.553+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9e8014420 con 0x7ff9f8060280 2024-01-21T20:04:56.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.553+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f8045ad0 con 0x7ff9f8060280 2024-01-21T20:04:56.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.553+0000 7ff9f4ff9700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9e402ff60 0x7ff9e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:56.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.553+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9e8041110 con 0x7ff9f8060280 2024-01-21T20:04:56.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.553+0000 7ff9eeffd700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9e402ff60 0x7ff9e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:56.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.554+0000 7ff9eeffd700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9e402ff60 0x7ff9e4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff9e00096a0 tx=0x7ff9e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:56.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.558+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9e80095f0 con 0x7ff9f8060280 2024-01-21T20:04:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.764+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9f8108110 con 0x7ff9f8060280 2024-01-21T20:04:56.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.765+0000 7ff9f4ff9700 1 -- 172.21.15.153:0/2850327099 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff9e801cb10 con 0x7ff9f8060280 2024-01-21T20:04:56.756 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:56.756 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9e402ff60 msgr2=0x7ff9e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9e402ff60 0x7ff9e4032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff9e00096a0 tx=0x7ff9e0006b40).stop 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 msgr2=0x7ff9f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f81066e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ff9e8013510 tx=0x7ff9e800b040).stop 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 shutdown_connections 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff9e402ff60 0x7ff9e4032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 --2- 172.21.15.153:0/2850327099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9f8060280 0x7ff9f81066e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 >> 172.21.15.153:0/2850327099 conn(0x7ff9f80f0b70 msgr2=0x7ff9f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 shutdown_connections 2024-01-21T20:04:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:56.767+0000 7ff9fdea7700 1 -- 172.21.15.153:0/2850327099 wait complete. 2024-01-21T20:04:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:04:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:56.695945+0000 mgr.smithi119.rqacbm (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:56.766407+0000 mon.smithi119 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.153:0/2850327099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:04:58.153 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:04:58.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:04:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:04:58.696277+0000 mgr.smithi119.rqacbm (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:04:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:58.784422+0000 mon.smithi119 (mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:04:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:58.784677+0000 mon.smithi119 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:04:59.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/3952083047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 msgr2=0x7fa0b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 --2- 172.21.15.153:0/3952083047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b80f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fa0a0009fb0 tx=0x7fa0a0009b30).stop 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/3952083047 shutdown_connections 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 --2- 172.21.15.153:0/3952083047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b80f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/3952083047 >> 172.21.15.153:0/3952083047 conn(0x7fa0b80f0b70 msgr2=0x7fa0b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/3952083047 shutdown_connections 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.339+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/3952083047 wait complete. 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0bcf0d700 1 Processor -- start 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0bcf0d700 1 -- start start 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0bcf0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:59.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0bcf0d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8105b20 con 0x7fa0b80f5410 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0b659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0b659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:55864/0 (socket says 172.21.15.153:55864) 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0b659c700 1 -- 172.21.15.153:0/1103614234 learned_addr learned my addr 172.21.15.153:0/1103614234 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.340+0000 7fa0b659c700 1 -- 172.21.15.153:0/1103614234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a0005040 con 0x7fa0b80f5410 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.341+0000 7fa0b659c700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b8105630 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fa0a0013510 tx=0x7fa0a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.341+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a0009320 con 0x7fa0b80f5410 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.341+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b8105d20 con 0x7fa0b80f5410 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.341+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0a0013f00 con 0x7fa0b80f5410 2024-01-21T20:04:59.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.341+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a0014d90 con 0x7fa0b80f5410 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.341+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b81061b0 con 0x7fa0b80f5410 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.342+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0a00095f0 con 0x7fa0b80f5410 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.342+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b8045ad0 con 0x7fa0b80f5410 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.342+0000 7fa0aeffd700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0a402ff60 0x7fa0a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.342+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0a0041000 con 0x7fa0b80f5410 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.342+0000 7fa0b5d9b700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0a402ff60 0x7fa0a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:04:59.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.342+0000 7fa0b5d9b700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0a402ff60 0x7fa0a4032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fa0a80096a0 tx=0x7fa0a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:04:59.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.346+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0a0014420 con 0x7fa0b80f5410 2024-01-21T20:04:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.553+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0b80fae60 con 0x7fa0b80f5410 2024-01-21T20:04:59.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.553+0000 7fa0aeffd700 1 -- 172.21.15.153:0/1103614234 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa0a0010230 con 0x7fa0b80f5410 2024-01-21T20:04:59.544 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:04:59.545 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0a402ff60 msgr2=0x7fa0a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0a402ff60 0x7fa0a4032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fa0a80096a0 tx=0x7fa0a8006b40).stop 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 msgr2=0x7fa0b8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b8105630 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fa0a0013510 tx=0x7fa0a000b040).stop 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 shutdown_connections 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0a402ff60 0x7fa0a4032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 --2- 172.21.15.153:0/1103614234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b80f5410 0x7fa0b8105630 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 >> 172.21.15.153:0/1103614234 conn(0x7fa0b80f0b70 msgr2=0x7fa0b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 shutdown_connections 2024-01-21T20:04:59.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:04:59.555+0000 7fa0bcf0d700 1 -- 172.21.15.153:0/1103614234 wait complete. 2024-01-21T20:04:59.547 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:00.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:04:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:04:59.554812+0000 mon.smithi119 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.153:0/1103614234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:00.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:00.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:01.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:00.696581+0000 mgr.smithi119.rqacbm (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/4216010030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 msgr2=0x7f8a180f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 --2- 172.21.15.153:0/4216010030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a180f6ea0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f8a0c009fb0 tx=0x7f8a0c009b30).stop 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/4216010030 shutdown_connections 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 --2- 172.21.15.153:0/4216010030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a180f6ea0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/4216010030 >> 172.21.15.153:0/4216010030 conn(0x7f8a180f02e0 msgr2=0x7f8a180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/4216010030 shutdown_connections 2024-01-21T20:05:02.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.111+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/4216010030 wait complete. 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.112+0000 7f8a1d4c5700 1 Processor -- start 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.112+0000 7f8a1d4c5700 1 -- start start 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.112+0000 7f8a1d4c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a18109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.112+0000 7f8a1d4c5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a180f5330 con 0x7f8a180f6a80 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.112+0000 7f8a16ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a18109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.112+0000 7f8a16ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a18109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:55890/0 (socket says 172.21.15.153:55890) 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f8a16ffd700 1 -- 172.21.15.153:0/2559563927 learned_addr learned my addr 172.21.15.153:0/2559563927 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f8a16ffd700 1 -- 172.21.15.153:0/2559563927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c005040 con 0x7f8a180f6a80 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f8a16ffd700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a18109410 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8a0c013510 tx=0x7f8a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a0c009320 con 0x7f8a180f6a80 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a180f5530 con 0x7f8a180f6a80 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a0c013f00 con 0x7f8a180f6a80 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a0c014dc0 con 0x7f8a180f6a80 2024-01-21T20:05:02.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.113+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a18109ac0 con 0x7f8a180f6a80 2024-01-21T20:05:02.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.114+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a0c0095f0 con 0x7f8a180f6a80 2024-01-21T20:05:02.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.114+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a18045ad0 con 0x7f8a180f6a80 2024-01-21T20:05:02.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.114+0000 7f89ff7fe700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8a0002ff60 0x7f8a00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:02.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.114+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a0c041040 con 0x7f8a180f6a80 2024-01-21T20:05:02.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.114+0000 7f8a167fc700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8a0002ff60 0x7f8a00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:02.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.115+0000 7f8a167fc700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8a0002ff60 0x7f8a00032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8a080096a0 tx=0x7f8a08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:02.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.119+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a0c014420 con 0x7f8a180f6a80 2024-01-21T20:05:02.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.322+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a1810a2e0 con 0x7f8a180f6a80 2024-01-21T20:05:02.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.323+0000 7f89ff7fe700 1 -- 172.21.15.153:0/2559563927 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a0c010230 con 0x7f8a180f6a80 2024-01-21T20:05:02.315 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:02.315 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8a0002ff60 msgr2=0x7f8a00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8a0002ff60 0x7f8a00032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8a080096a0 tx=0x7f8a08006b40).stop 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 msgr2=0x7f8a18109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a18109410 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f8a0c013510 tx=0x7f8a0c00b040).stop 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 shutdown_connections 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8a0002ff60 0x7f8a00032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 --2- 172.21.15.153:0/2559563927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a180f6a80 0x7f8a18109410 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:02.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 >> 172.21.15.153:0/2559563927 conn(0x7f8a180f02e0 msgr2=0x7f8a180f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:02.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 shutdown_connections 2024-01-21T20:05:02.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:02.325+0000 7f8a1d4c5700 1 -- 172.21.15.153:0/2559563927 wait complete. 2024-01-21T20:05:02.317 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:02.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:02.324624+0000 mon.smithi119 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.153:0/2559563927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:03.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:03.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:03.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:02.696897+0000 mgr.smithi119.rqacbm (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:04.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 -- 172.21.15.153:0/2954242969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 msgr2=0x7fcbc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:04.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 --2- 172.21.15.153:0/2954242969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc00f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fcbb0009fb0 tx=0x7fcbb0009b30).stop 2024-01-21T20:05:04.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 -- 172.21.15.153:0/2954242969 shutdown_connections 2024-01-21T20:05:04.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 --2- 172.21.15.153:0/2954242969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc00f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:04.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 -- 172.21.15.153:0/2954242969 >> 172.21.15.153:0/2954242969 conn(0x7fcbc00f0b70 msgr2=0x7fcbc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:04.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 -- 172.21.15.153:0/2954242969 shutdown_connections 2024-01-21T20:05:04.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.871+0000 7fcbc5184700 1 -- 172.21.15.153:0/2954242969 wait complete. 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbc5184700 1 Processor -- start 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbc5184700 1 -- start start 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbc5184700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbc5184700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc0107240 con 0x7fcbc00f5410 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbbed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbbed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:58702/0 (socket says 172.21.15.153:58702) 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.872+0000 7fcbbed9d700 1 -- 172.21.15.153:0/2505771099 learned_addr learned my addr 172.21.15.153:0/2505771099 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:04.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbbed9d700 1 -- 172.21.15.153:0/2505771099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb0005040 con 0x7fcbc00f5410 2024-01-21T20:05:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbbed9d700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc010b850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fcbb0013510 tx=0x7fcbb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb0009320 con 0x7fcbc00f5410 2024-01-21T20:05:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbb0013f00 con 0x7fcbc00f5410 2024-01-21T20:05:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc0060220 con 0x7fcbc00f5410 2024-01-21T20:05:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbb0014d90 con 0x7fcbc00f5410 2024-01-21T20:05:04.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.873+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc010bd40 con 0x7fcbc00f5410 2024-01-21T20:05:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.874+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbb00095f0 con 0x7fcbc00f5410 2024-01-21T20:05:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.874+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc0045ad0 con 0x7fcbc00f5410 2024-01-21T20:05:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.874+0000 7fcbb77fe700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcbac02ff60 0x7fcbac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.874+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbb0041000 con 0x7fcbc00f5410 2024-01-21T20:05:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.874+0000 7fcbbe59c700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcbac02ff60 0x7fcbac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.874+0000 7fcbbe59c700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcbac02ff60 0x7fcbac032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fcba80096a0 tx=0x7fcba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:04.878+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbb0014420 con 0x7fcbc00f5410 2024-01-21T20:05:05.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.084+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbc00fae60 con 0x7fcbc00f5410 2024-01-21T20:05:05.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.085+0000 7fcbb77fe700 1 -- 172.21.15.153:0/2505771099 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcbb0010230 con 0x7fcbc00f5410 2024-01-21T20:05:05.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:05.076 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:05.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcbac02ff60 msgr2=0x7fcbac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcbac02ff60 0x7fcbac032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fcba80096a0 tx=0x7fcba8006b40).stop 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 msgr2=0x7fcbc010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc010b850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fcbb0013510 tx=0x7fcbb000b040).stop 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 shutdown_connections 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fcbac02ff60 0x7fcbac032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 --2- 172.21.15.153:0/2505771099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc00f5410 0x7fcbc010b850 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 >> 172.21.15.153:0/2505771099 conn(0x7fcbc00f0b70 msgr2=0x7fcbc00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 shutdown_connections 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:05.086+0000 7fcbc5184700 1 -- 172.21.15.153:0/2505771099 wait complete. 2024-01-21T20:05:05.078 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:05.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:04.697237+0000 mgr.smithi119.rqacbm (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:06.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:05.086098+0000 mon.smithi119 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.153:0/2505771099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:06.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:06.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:06.697557+0000 mgr.smithi119.rqacbm (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:07.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.661+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/2885360438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 msgr2=0x7fa7c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:07.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.661+0000 7fa7ca5cb700 1 --2- 172.21.15.153:0/2885360438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c40f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fa7b4009fb0 tx=0x7fa7b4009b30).stop 2024-01-21T20:05:07.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.661+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/2885360438 shutdown_connections 2024-01-21T20:05:07.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.661+0000 7fa7ca5cb700 1 --2- 172.21.15.153:0/2885360438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c40f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:07.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.661+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/2885360438 >> 172.21.15.153:0/2885360438 conn(0x7fa7c40f1c70 msgr2=0x7fa7c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:07.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.661+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/2885360438 shutdown_connections 2024-01-21T20:05:07.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.662+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/2885360438 wait complete. 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.662+0000 7fa7ca5cb700 1 Processor -- start 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.662+0000 7fa7ca5cb700 1 -- start start 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7ca5cb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7ca5cb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4106c20 con 0x7fa7c40f6510 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7c3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7c3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:58714/0 (socket says 172.21.15.153:58714) 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7c3fff700 1 -- 172.21.15.153:0/1916804287 learned_addr learned my addr 172.21.15.153:0/1916804287 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7c3fff700 1 -- 172.21.15.153:0/1916804287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b4005040 con 0x7fa7c40f6510 2024-01-21T20:05:07.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7c3fff700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c4106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fa7b4013510 tx=0x7fa7b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.663+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7b4009320 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c4106e20 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c41072b0 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7b4013f00 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7b4014d60 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7b40263a0 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c4045ad0 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.664+0000 7fa7c0ff9700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa7b002ff60 0x7fa7b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.665+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7b40413d0 con 0x7fa7c40f6510 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.665+0000 7fa7c37fe700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa7b002ff60 0x7fa7b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:07.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.665+0000 7fa7c37fe700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa7b002ff60 0x7fa7b0032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa7ac0096a0 tx=0x7fa7ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:07.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.669+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7b401d700 con 0x7fa7c40f6510 2024-01-21T20:05:07.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.876+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7c40fbf60 con 0x7fa7c40f6510 2024-01-21T20:05:07.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.877+0000 7fa7c0ff9700 1 -- 172.21.15.153:0/1916804287 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa7b4012330 con 0x7fa7c40f6510 2024-01-21T20:05:07.868 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:07.868 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:07.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa7b002ff60 msgr2=0x7fa7b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:07.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa7b002ff60 0x7fa7b0032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa7ac0096a0 tx=0x7fa7ac006b40).stop 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 msgr2=0x7fa7c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c4106730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fa7b4013510 tx=0x7fa7b400b040).stop 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 shutdown_connections 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa7b002ff60 0x7fa7b0032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 --2- 172.21.15.153:0/1916804287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7c40f6510 0x7fa7c4106730 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 >> 172.21.15.153:0/1916804287 conn(0x7fa7c40f1c70 msgr2=0x7fa7c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 shutdown_connections 2024-01-21T20:05:07.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:07.878+0000 7fa7ca5cb700 1 -- 172.21.15.153:0/1916804287 wait complete. 2024-01-21T20:05:07.871 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:08.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:07.878337+0000 mon.smithi119 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.153:0/1916804287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:09.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:09.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:05:08.357822+0000 mgr.smithi119.rqacbm (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi119 ... 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:00:46Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:01:24Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:02:01Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:05:09.338 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:03:15Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:03:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:04:30Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Traceback (most recent call last): 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: main() 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: r = ctx.func(ctx) 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: return func(ctx) 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T20:05:09.339 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: raise RuntimeError('uid/gid not found') 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: RuntimeError: uid/gid not found 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Traceback (most recent call last): 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: yield (conn, connr) 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: code, '\n'.join(err))) 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi119 ... 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:05:09.340 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:00:46Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:01:24Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:02:01Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi119 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:03:15Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:03:53Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr time="2024-01-21T20:04:30Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Traceback (most recent call last): 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: main() 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: r = ctx.func(ctx) 2024-01-21T20:05:09.341 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: return func(ctx) 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: raise RuntimeError('uid/gid not found') 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: RuntimeError: uid/gid not found 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:08.359050+0000 mon.smithi119 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi119", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:08.361769+0000 mon.smithi119 (mon.0) 227 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi119", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:08.362592+0000 mon.smithi119 (mon.0) 228 : audit [DBG] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:05:08.363360+0000 mgr.smithi119.rqacbm (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi119 on smithi119 2024-01-21T20:05:09.342 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:08.697824+0000 mgr.smithi119.rqacbm (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:10.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.464+0000 7fbc135e9700 1 -- 172.21.15.153:0/3698508179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 msgr2=0x7fbc0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.464+0000 7fbc135e9700 1 --2- 172.21.15.153:0/3698508179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fbc08009fb0 tx=0x7fbc08009b30).stop 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 -- 172.21.15.153:0/3698508179 shutdown_connections 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 --2- 172.21.15.153:0/3698508179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 -- 172.21.15.153:0/3698508179 >> 172.21.15.153:0/3698508179 conn(0x7fbc0c0f0b70 msgr2=0x7fbc0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 -- 172.21.15.153:0/3698508179 shutdown_connections 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 -- 172.21.15.153:0/3698508179 wait complete. 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 Processor -- start 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 -- start start 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc135e9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c0fb820 con 0x7fbc0c0f5410 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc11385700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc11385700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:58734/0 (socket says 172.21.15.153:58734) 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.465+0000 7fbc11385700 1 -- 172.21.15.153:0/358622811 learned_addr learned my addr 172.21.15.153:0/358622811 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc11385700 1 -- 172.21.15.153:0/358622811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc08005040 con 0x7fbc0c0f5410 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc11385700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0fd3a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fbc08013510 tx=0x7fbc0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc08009320 con 0x7fbc0c0f5410 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c0fb9c0 con 0x7fbc0c0f5410 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc08013f00 con 0x7fbc0c0f5410 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc08014d90 con 0x7fbc0c0f5410 2024-01-21T20:05:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.466+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c0fbe50 con 0x7fbc0c0f5410 2024-01-21T20:05:10.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.467+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc08014420 con 0x7fbc0c0f5410 2024-01-21T20:05:10.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.467+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c059100 con 0x7fbc0c0f5410 2024-01-21T20:05:10.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.467+0000 7fbc01ffb700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbbf802ff60 0x7fbbf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.467+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc08041110 con 0x7fbc0c0f5410 2024-01-21T20:05:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.467+0000 7fbc10b84700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbbf802ff60 0x7fbbf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:10.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.468+0000 7fbc10b84700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbbf802ff60 0x7fbbf8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:10.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.471+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc080095f0 con 0x7fbc0c0f5410 2024-01-21T20:05:10.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.677+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc0c045ad0 con 0x7fbc0c0f5410 2024-01-21T20:05:10.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.678+0000 7fbc01ffb700 1 -- 172.21.15.153:0/358622811 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc0801db70 con 0x7fbc0c0f5410 2024-01-21T20:05:10.669 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:10.669 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbbf802ff60 msgr2=0x7fbbf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbbf802ff60 0x7fbbf8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).stop 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 msgr2=0x7fbc0c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0fd3a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fbc08013510 tx=0x7fbc0800b040).stop 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 shutdown_connections 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbbf802ff60 0x7fbbf8032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 --2- 172.21.15.153:0/358622811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc0c0f5410 0x7fbc0c0fd3a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 >> 172.21.15.153:0/358622811 conn(0x7fbc0c0f0b70 msgr2=0x7fbc0c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 shutdown_connections 2024-01-21T20:05:10.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:10.680+0000 7fbc135e9700 1 -- 172.21.15.153:0/358622811 wait complete. 2024-01-21T20:05:10.672 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:09.821490+0000 mon.smithi119 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:05:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:09.905995+0000 mon.smithi119 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:05:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:09.907812+0000 mon.smithi119 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:05:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:09.908229+0000 mon.smithi119 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T20:05:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:09.908658+0000 mgr.smithi119.rqacbm (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-21T20:05:11.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:09.910926+0000 mon.smithi119 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:05:11.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:05:09.913937+0000 mgr.smithi119.rqacbm (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi119 on smithi119 2024-01-21T20:05:11.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:10.679731+0000 mon.smithi119 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.153:0/358622811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:11.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:10.698171+0000 mgr.smithi119.rqacbm (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:12.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:12.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:13.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:12.698541+0000 mgr.smithi119.rqacbm (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 -- 172.21.15.153:0/3275235900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 msgr2=0x7f7f380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 --2- 172.21.15.153:0/3275235900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f380f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f7f28009fb0 tx=0x7f7f28009b30).stop 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 -- 172.21.15.153:0/3275235900 shutdown_connections 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 --2- 172.21.15.153:0/3275235900 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f380f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 -- 172.21.15.153:0/3275235900 >> 172.21.15.153:0/3275235900 conn(0x7f7f380f0b70 msgr2=0x7f7f380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 -- 172.21.15.153:0/3275235900 shutdown_connections 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 -- 172.21.15.153:0/3275235900 wait complete. 2024-01-21T20:05:13.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.279+0000 7f7f3e391700 1 Processor -- start 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f3e391700 1 -- start start 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f3e391700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f38109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f3e391700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f38060240 con 0x7f7f380f5410 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f38109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f37fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f38109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:58764/0 (socket says 172.21.15.153:58764) 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f37fff700 1 -- 172.21.15.153:0/370615068 learned_addr learned my addr 172.21.15.153:0/370615068 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f37fff700 1 -- 172.21.15.153:0/370615068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f28005040 con 0x7f7f380f5410 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.280+0000 7f7f37fff700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f38109430 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f7f28013510 tx=0x7f7f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f28009320 con 0x7f7f380f5410 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f28013f00 con 0x7f7f380f5410 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f38060440 con 0x7f7f380f5410 2024-01-21T20:05:13.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f28014d90 con 0x7f7f380f5410 2024-01-21T20:05:13.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f38109ae0 con 0x7f7f380f5410 2024-01-21T20:05:13.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f280095f0 con 0x7f7f380f5410 2024-01-21T20:05:13.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.281+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f38045ad0 con 0x7f7f380f5410 2024-01-21T20:05:13.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.282+0000 7f7f34ff9700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f2002ff60 0x7f7f20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:13.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.282+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f280410c0 con 0x7f7f380f5410 2024-01-21T20:05:13.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.282+0000 7f7f377fe700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f2002ff60 0x7f7f20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:13.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.282+0000 7f7f377fe700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f2002ff60 0x7f7f20032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7f2c0096a0 tx=0x7f7f2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:13.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.286+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f28014420 con 0x7f7f380f5410 2024-01-21T20:05:13.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.491+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f380fae60 con 0x7f7f380f5410 2024-01-21T20:05:13.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.492+0000 7f7f34ff9700 1 -- 172.21.15.153:0/370615068 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f2801d7f0 con 0x7f7f380f5410 2024-01-21T20:05:13.483 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:13.483 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:13.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f2002ff60 msgr2=0x7f7f20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:13.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f2002ff60 0x7f7f20032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f7f2c0096a0 tx=0x7f7f2c006b40).stop 2024-01-21T20:05:13.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 msgr2=0x7f7f38109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:13.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f38109430 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f7f28013510 tx=0x7f7f2800b040).stop 2024-01-21T20:05:13.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 shutdown_connections 2024-01-21T20:05:13.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7f2002ff60 0x7f7f20032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:13.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 --2- 172.21.15.153:0/370615068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7f380f5410 0x7f7f38109430 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:13.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 >> 172.21.15.153:0/370615068 conn(0x7f7f380f0b70 msgr2=0x7f7f380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:13.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 shutdown_connections 2024-01-21T20:05:13.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:13.494+0000 7f7f3e391700 1 -- 172.21.15.153:0/370615068 wait complete. 2024-01-21T20:05:13.489 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:14.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:13.493606+0000 mon.smithi119 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.153:0/370615068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:14.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:14.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:14.698844+0000 mgr.smithi119.rqacbm (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- 172.21.15.153:0/1006970378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 msgr2=0x7f9ed80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 --2- 172.21.15.153:0/1006970378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f4be0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f9ec8009fb0 tx=0x7f9ec8009b30).stop 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- 172.21.15.153:0/1006970378 shutdown_connections 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 --2- 172.21.15.153:0/1006970378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f4be0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- 172.21.15.153:0/1006970378 >> 172.21.15.153:0/1006970378 conn(0x7f9ed80f02e0 msgr2=0x7f9ed80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- 172.21.15.153:0/1006970378 shutdown_connections 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- 172.21.15.153:0/1006970378 wait complete. 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 Processor -- start 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- start start 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.087+0000 7f9edd813700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed80f57f0 con 0x7f9ed80f47c0 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ed6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ed6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:49046/0 (socket says 172.21.15.153:49046) 2024-01-21T20:05:16.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ed6ffd700 1 -- 172.21.15.153:0/2954749522 learned_addr learned my addr 172.21.15.153:0/2954749522 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ed6ffd700 1 -- 172.21.15.153:0/2954749522 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec8005040 con 0x7f9ed80f47c0 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ed6ffd700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f7370 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f9ec8013510 tx=0x7f9ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec8009320 con 0x7f9ed80f47c0 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed80f5990 con 0x7f9ed80f47c0 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ec8013f00 con 0x7f9ed80f47c0 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec8014d90 con 0x7f9ed80f47c0 2024-01-21T20:05:16.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.088+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed80f5e20 con 0x7f9ed80f47c0 2024-01-21T20:05:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.089+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ec80095f0 con 0x7f9ed80f47c0 2024-01-21T20:05:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.089+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed8059100 con 0x7f9ed80f47c0 2024-01-21T20:05:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.089+0000 7f9ebf7fe700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9ec002ffb0 0x7f9ec0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.089+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ec80410c0 con 0x7f9ed80f47c0 2024-01-21T20:05:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.089+0000 7f9ed67fc700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9ec002ffb0 0x7f9ec0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:16.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.090+0000 7f9ed67fc700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9ec002ffb0 0x7f9ec0032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9ecc0096a0 tx=0x7f9ecc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:16.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.094+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ec8014420 con 0x7f9ed80f47c0 2024-01-21T20:05:16.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.300+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ed80fbf60 con 0x7f9ed80f47c0 2024-01-21T20:05:16.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.300+0000 7f9ebf7fe700 1 -- 172.21.15.153:0/2954749522 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ec8010230 con 0x7f9ed80f47c0 2024-01-21T20:05:16.291 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:16.291 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:16.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9ec002ffb0 msgr2=0x7f9ec0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:16.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9ec002ffb0 0x7f9ec0032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9ecc0096a0 tx=0x7f9ecc006b40).stop 2024-01-21T20:05:16.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 msgr2=0x7f9ed80f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:16.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f7370 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f9ec8013510 tx=0x7f9ec800b040).stop 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 shutdown_connections 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9ec002ffb0 0x7f9ec0032470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 --2- 172.21.15.153:0/2954749522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ed80f47c0 0x7f9ed80f7370 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 >> 172.21.15.153:0/2954749522 conn(0x7f9ed80f02e0 msgr2=0x7f9ed80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 shutdown_connections 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:16.302+0000 7f9edd813700 1 -- 172.21.15.153:0/2954749522 wait complete. 2024-01-21T20:05:16.294 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:16.301806+0000 mon.smithi119 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.153:0/2954749522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:16.699109+0000 mgr.smithi119.rqacbm (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:17.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:17.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:18.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 -- 172.21.15.153:0/306878688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 msgr2=0x7f78480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:18.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 --2- 172.21.15.153:0/306878688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f78480f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f783c009fb0 tx=0x7f783c009b30).stop 2024-01-21T20:05:18.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 -- 172.21.15.153:0/306878688 shutdown_connections 2024-01-21T20:05:18.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 --2- 172.21.15.153:0/306878688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f78480f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:18.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 -- 172.21.15.153:0/306878688 >> 172.21.15.153:0/306878688 conn(0x7f78480f0b70 msgr2=0x7f78480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:18.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 -- 172.21.15.153:0/306878688 shutdown_connections 2024-01-21T20:05:18.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.883+0000 7f784d676700 1 -- 172.21.15.153:0/306878688 wait complete. 2024-01-21T20:05:18.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f784d676700 1 Processor -- start 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f784d676700 1 -- start start 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f784d676700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f7848106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f784d676700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7848106bf0 con 0x7f78480f5410 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f7846ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f7848106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f7846ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f7848106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:49056/0 (socket says 172.21.15.153:49056) 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f7846ffd700 1 -- 172.21.15.153:0/2454579119 learned_addr learned my addr 172.21.15.153:0/2454579119 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f7846ffd700 1 -- 172.21.15.153:0/2454579119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c005040 con 0x7f78480f5410 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f7846ffd700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f7848106700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f783c013510 tx=0x7f783c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f783c009320 con 0x7f78480f5410 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f783c013f00 con 0x7f78480f5410 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f783c014d90 con 0x7f78480f5410 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.884+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7848106df0 con 0x7f78480f5410 2024-01-21T20:05:18.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.885+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7848107280 con 0x7f78480f5410 2024-01-21T20:05:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.885+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f783c014420 con 0x7f78480f5410 2024-01-21T20:05:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.885+0000 7f782f7fe700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f783002ffb0 0x7f7830032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.885+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7848045ad0 con 0x7f78480f5410 2024-01-21T20:05:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.885+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f783c041090 con 0x7f78480f5410 2024-01-21T20:05:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.886+0000 7f78467fc700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f783002ffb0 0x7f7830032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.886+0000 7f78467fc700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f783002ffb0 0x7f7830032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f78380096a0 tx=0x7f7838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:18.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:18.890+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f783c01dd50 con 0x7f78480f5410 2024-01-21T20:05:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.095+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78480faf50 con 0x7f78480f5410 2024-01-21T20:05:19.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.096+0000 7f782f7fe700 1 -- 172.21.15.153:0/2454579119 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f783c023250 con 0x7f78480f5410 2024-01-21T20:05:19.087 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:19.087 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:19.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:18.699328+0000 mgr.smithi119.rqacbm (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:19.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f783002ffb0 msgr2=0x7f7830032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:19.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f783002ffb0 0x7f7830032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f78380096a0 tx=0x7f7838006b40).stop 2024-01-21T20:05:19.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 msgr2=0x7f7848106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:19.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f7848106700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f783c013510 tx=0x7f783c00b040).stop 2024-01-21T20:05:19.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 shutdown_connections 2024-01-21T20:05:19.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f783002ffb0 0x7f7830032470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:19.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 --2- 172.21.15.153:0/2454579119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480f5410 0x7f7848106700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:19.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 >> 172.21.15.153:0/2454579119 conn(0x7f78480f0b70 msgr2=0x7f78480f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:19.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 shutdown_connections 2024-01-21T20:05:19.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:19.098+0000 7f784d676700 1 -- 172.21.15.153:0/2454579119 wait complete. 2024-01-21T20:05:19.090 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:20.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:19.097535+0000 mon.smithi119 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.153:0/2454579119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:20.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:20.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:21.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:20.699578+0000 mgr.smithi119.rqacbm (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 -- 172.21.15.153:0/2000618967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 msgr2=0x7fef240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 --2- 172.21.15.153:0/2000618967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef240f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fef18009fb0 tx=0x7fef18009b30).stop 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 -- 172.21.15.153:0/2000618967 shutdown_connections 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 --2- 172.21.15.153:0/2000618967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef240f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 -- 172.21.15.153:0/2000618967 >> 172.21.15.153:0/2000618967 conn(0x7fef240f0b70 msgr2=0x7fef240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 -- 172.21.15.153:0/2000618967 shutdown_connections 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.622+0000 7fef2b151700 1 -- 172.21.15.153:0/2000618967 wait complete. 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef2b151700 1 Processor -- start 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef2b151700 1 -- start start 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef2b151700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef24105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:21.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef2b151700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef24105b20 con 0x7fef240f5410 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef28eed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef24105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef28eed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef24105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:49076/0 (socket says 172.21.15.153:49076) 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef28eed700 1 -- 172.21.15.153:0/3994947280 learned_addr learned my addr 172.21.15.153:0/3994947280 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.623+0000 7fef28eed700 1 -- 172.21.15.153:0/3994947280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef18005040 con 0x7fef240f5410 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.624+0000 7fef28eed700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef24105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fef18013510 tx=0x7fef1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.624+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef18009320 con 0x7fef240f5410 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.624+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef24105d20 con 0x7fef240f5410 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.624+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef18013f00 con 0x7fef240f5410 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.624+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef18014d90 con 0x7fef240f5410 2024-01-21T20:05:21.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.624+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef241061b0 con 0x7fef240f5410 2024-01-21T20:05:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.625+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef180263a0 con 0x7fef240f5410 2024-01-21T20:05:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.625+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef24045ad0 con 0x7fef240f5410 2024-01-21T20:05:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.625+0000 7fef217fa700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fef1402ffb0 0x7fef14032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.625+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef18041040 con 0x7fef240f5410 2024-01-21T20:05:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.625+0000 7fef23fff700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fef1402ffb0 0x7fef14032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:21.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.625+0000 7fef23fff700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fef1402ffb0 0x7fef14032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fef100096a0 tx=0x7fef10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:21.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.631+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef180126f0 con 0x7fef240f5410 2024-01-21T20:05:21.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.835+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef240fae60 con 0x7fef240f5410 2024-01-21T20:05:21.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.836+0000 7fef217fa700 1 -- 172.21.15.153:0/3994947280 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fef18012330 con 0x7fef240f5410 2024-01-21T20:05:21.827 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:21.827 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:21.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fef1402ffb0 msgr2=0x7fef14032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:21.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fef1402ffb0 0x7fef14032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fef100096a0 tx=0x7fef10006b40).stop 2024-01-21T20:05:21.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 msgr2=0x7fef24105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef24105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fef18013510 tx=0x7fef1800b040).stop 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 shutdown_connections 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fef1402ffb0 0x7fef14032470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 --2- 172.21.15.153:0/3994947280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fef240f5410 0x7fef24105630 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 >> 172.21.15.153:0/3994947280 conn(0x7fef240f0b70 msgr2=0x7fef240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 shutdown_connections 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:21.837+0000 7fef2b151700 1 -- 172.21.15.153:0/3994947280 wait complete. 2024-01-21T20:05:21.829 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:22.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:21.837132+0000 mon.smithi119 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.153:0/3994947280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:23.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:23.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:23.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:22.699815+0000 mgr.smithi119.rqacbm (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 -- 172.21.15.153:0/2267735926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 msgr2=0x7fb0140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 --2- 172.21.15.153:0/2267735926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0140f90f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fb008009fb0 tx=0x7fb008009b30).stop 2024-01-21T20:05:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 -- 172.21.15.153:0/2267735926 shutdown_connections 2024-01-21T20:05:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 --2- 172.21.15.153:0/2267735926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0140f90f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 -- 172.21.15.153:0/2267735926 >> 172.21.15.153:0/2267735926 conn(0x7fb0140f0b70 msgr2=0x7fb0140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 -- 172.21.15.153:0/2267735926 shutdown_connections 2024-01-21T20:05:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.411+0000 7fb01afd2700 1 -- 172.21.15.153:0/2267735926 wait complete. 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb01afd2700 1 Processor -- start 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb01afd2700 1 -- start start 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb01afd2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0141066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb01afd2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014106bd0 con 0x7fb014060280 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb018d6e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0141066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb018d6e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0141066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47848/0 (socket says 172.21.15.153:47848) 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.412+0000 7fb018d6e700 1 -- 172.21.15.153:0/3027550233 learned_addr learned my addr 172.21.15.153:0/3027550233 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb018d6e700 1 -- 172.21.15.153:0/3027550233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008005040 con 0x7fb014060280 2024-01-21T20:05:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb018d6e700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0141066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb008013510 tx=0x7fb00800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb008009320 con 0x7fb014060280 2024-01-21T20:05:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb014106dd0 con 0x7fb014060280 2024-01-21T20:05:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb008013f00 con 0x7fb014060280 2024-01-21T20:05:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb008014dc0 con 0x7fb014060280 2024-01-21T20:05:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.413+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb014107260 con 0x7fb014060280 2024-01-21T20:05:24.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.414+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb008014420 con 0x7fb014060280 2024-01-21T20:05:24.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.414+0000 7fb0117fa700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb00402ff60 0x7fb004032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:24.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.414+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb014045ad0 con 0x7fb014060280 2024-01-21T20:05:24.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.414+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb008041110 con 0x7fb014060280 2024-01-21T20:05:24.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.414+0000 7fb013fff700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb00402ff60 0x7fb004032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:24.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.415+0000 7fb013fff700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb00402ff60 0x7fb004032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fb0000096a0 tx=0x7fb000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.419+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0080095f0 con 0x7fb014060280 2024-01-21T20:05:24.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.623+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb014108110 con 0x7fb014060280 2024-01-21T20:05:24.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.624+0000 7fb0117fa700 1 -- 172.21.15.153:0/3027550233 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb00802c090 con 0x7fb014060280 2024-01-21T20:05:24.614 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:24.614 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb00402ff60 msgr2=0x7fb004032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb00402ff60 0x7fb004032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fb0000096a0 tx=0x7fb000006b40).stop 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 msgr2=0x7fb0141066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0141066e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fb008013510 tx=0x7fb00800b040).stop 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 shutdown_connections 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb00402ff60 0x7fb004032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 --2- 172.21.15.153:0/3027550233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb014060280 0x7fb0141066e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 >> 172.21.15.153:0/3027550233 conn(0x7fb0140f0b70 msgr2=0x7fb0140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 shutdown_connections 2024-01-21T20:05:24.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:24.626+0000 7fb01afd2700 1 -- 172.21.15.153:0/3027550233 wait complete. 2024-01-21T20:05:24.617 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:24.925 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:24.624684+0000 mon.smithi119 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.153:0/3027550233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:25.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:24.700044+0000 mgr.smithi119.rqacbm (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:26.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:26.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:27.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 -- 172.21.15.153:0/881128905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 msgr2=0x7f0f240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:27.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 --2- 172.21.15.153:0/881128905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f240f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f0f18009fb0 tx=0x7f0f18009b30).stop 2024-01-21T20:05:27.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 -- 172.21.15.153:0/881128905 shutdown_connections 2024-01-21T20:05:27.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 --2- 172.21.15.153:0/881128905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f240f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 -- 172.21.15.153:0/881128905 >> 172.21.15.153:0/881128905 conn(0x7f0f240f1c70 msgr2=0x7f0f240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 -- 172.21.15.153:0/881128905 shutdown_connections 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.216+0000 7f0f29438700 1 -- 172.21.15.153:0/881128905 wait complete. 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.217+0000 7f0f29438700 1 Processor -- start 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.217+0000 7f0f29438700 1 -- start start 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.217+0000 7f0f29438700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.217+0000 7f0f29438700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24106c20 con 0x7f0f240f6510 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f22ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f22ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47862/0 (socket says 172.21.15.153:47862) 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f22ffd700 1 -- 172.21.15.153:0/1292417092 learned_addr learned my addr 172.21.15.153:0/1292417092 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f22ffd700 1 -- 172.21.15.153:0/1292417092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18005040 con 0x7f0f240f6510 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f22ffd700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f24106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0f18013510 tx=0x7f0f1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f18009320 con 0x7f0f240f6510 2024-01-21T20:05:27.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f24106e20 con 0x7f0f240f6510 2024-01-21T20:05:27.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f18013f00 con 0x7f0f240f6510 2024-01-21T20:05:27.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f18014d90 con 0x7f0f240f6510 2024-01-21T20:05:27.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.218+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f241072b0 con 0x7f0f240f6510 2024-01-21T20:05:27.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.219+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f180263a0 con 0x7f0f240f6510 2024-01-21T20:05:27.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.219+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f24045ad0 con 0x7f0f240f6510 2024-01-21T20:05:27.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.219+0000 7f0f0b7fe700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f0c02ff60 0x7f0f0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:27.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.219+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f180410d0 con 0x7f0f240f6510 2024-01-21T20:05:27.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.219+0000 7f0f227fc700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f0c02ff60 0x7f0f0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:27.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.220+0000 7f0f227fc700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f0c02ff60 0x7f0f0c032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0f140096a0 tx=0x7f0f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:27.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.224+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f180126f0 con 0x7f0f240f6510 2024-01-21T20:05:27.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.430+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f240fbf60 con 0x7f0f240f6510 2024-01-21T20:05:27.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.431+0000 7f0f0b7fe700 1 -- 172.21.15.153:0/1292417092 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f18012330 con 0x7f0f240f6510 2024-01-21T20:05:27.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:27.421 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:27.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f0c02ff60 msgr2=0x7f0f0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f0c02ff60 0x7f0f0c032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0f140096a0 tx=0x7f0f14006b40).stop 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 msgr2=0x7f0f24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f24106730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0f18013510 tx=0x7f0f1800b040).stop 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 shutdown_connections 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0f0c02ff60 0x7f0f0c032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 --2- 172.21.15.153:0/1292417092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f240f6510 0x7f0f24106730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.432+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 >> 172.21.15.153:0/1292417092 conn(0x7f0f240f1c70 msgr2=0x7f0f240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.433+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 shutdown_connections 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:27.433+0000 7f0f29438700 1 -- 172.21.15.153:0/1292417092 wait complete. 2024-01-21T20:05:27.423 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:26.307261+0000 mon.smithi119 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' 2024-01-21T20:05:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:05:26.308451+0000 mgr.smithi119.rqacbm (mgr.14162) 169 : cephadm [INF] Deploying daemon node-exporter.smithi119 on smithi119 2024-01-21T20:05:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:26.704543+0000 mgr.smithi119.rqacbm (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:28.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:27.431361+0000 mon.smithi119 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.153:0/1292417092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:28.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:28.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:29.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:28.705009+0000 mgr.smithi119.rqacbm (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 -- 172.21.15.153:0/3074355616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 msgr2=0x7f98200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 --2- 172.21.15.153:0/3074355616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f98200f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f9810009fb0 tx=0x7f9810009b30).stop 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 -- 172.21.15.153:0/3074355616 shutdown_connections 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 --2- 172.21.15.153:0/3074355616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f98200f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 -- 172.21.15.153:0/3074355616 >> 172.21.15.153:0/3074355616 conn(0x7f98200f0b70 msgr2=0x7f98200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 -- 172.21.15.153:0/3074355616 shutdown_connections 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.112+0000 7f98276fd700 1 -- 172.21.15.153:0/3074355616 wait complete. 2024-01-21T20:05:30.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f98276fd700 1 Processor -- start 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f98276fd700 1 -- start start 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f98276fd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f9820105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f98276fd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820105b20 con 0x7f98200f5410 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f9825499700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f9820105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f9825499700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f9820105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47872/0 (socket says 172.21.15.153:47872) 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.113+0000 7f9825499700 1 -- 172.21.15.153:0/1933092695 learned_addr learned my addr 172.21.15.153:0/1933092695 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f9825499700 1 -- 172.21.15.153:0/1933092695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810005040 con 0x7f98200f5410 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f9825499700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f9820105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9810014000 tx=0x7f981000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98100095f0 con 0x7f98200f5410 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9810007e70 con 0x7f98200f5410 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9820105d20 con 0x7f98200f5410 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9810014ea0 con 0x7f98200f5410 2024-01-21T20:05:30.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.114+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98201061b0 con 0x7f98200f5410 2024-01-21T20:05:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.115+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9810025070 con 0x7f98200f5410 2024-01-21T20:05:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.115+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820045ad0 con 0x7f98200f5410 2024-01-21T20:05:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.115+0000 7f9815ffb700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f980c02ff60 0x7f980c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:30.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.115+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9810040970 con 0x7f98200f5410 2024-01-21T20:05:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.116+0000 7f9824c98700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f980c02ff60 0x7f980c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:30.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.116+0000 7f9824c98700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f980c02ff60 0x7f980c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f981c0096a0 tx=0x7f981c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:30.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.120+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98100079c0 con 0x7f98200f5410 2024-01-21T20:05:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.324+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98200fae60 con 0x7f98200f5410 2024-01-21T20:05:30.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.325+0000 7f9815ffb700 1 -- 172.21.15.153:0/1933092695 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f981000f300 con 0x7f98200f5410 2024-01-21T20:05:30.315 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:30.315 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f980c02ff60 msgr2=0x7f980c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f980c02ff60 0x7f980c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f981c0096a0 tx=0x7f981c006b40).stop 2024-01-21T20:05:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 msgr2=0x7f9820105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f9820105630 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9810014000 tx=0x7f981000b040).stop 2024-01-21T20:05:30.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 shutdown_connections 2024-01-21T20:05:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f980c02ff60 0x7f980c032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 --2- 172.21.15.153:0/1933092695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98200f5410 0x7f9820105630 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 >> 172.21.15.153:0/1933092695 conn(0x7f98200f0b70 msgr2=0x7f98200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 shutdown_connections 2024-01-21T20:05:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:30.327+0000 7f98276fd700 1 -- 172.21.15.153:0/1933092695 wait complete. 2024-01-21T20:05:30.318 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:30.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:30.325395+0000 mon.smithi119 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.153:0/1933092695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:31.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:31.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:31.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:30.705369+0000 mgr.smithi119.rqacbm (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.927+0000 7fb6283d7700 1 -- 172.21.15.153:0/3437189262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 msgr2=0x7fb6200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.927+0000 7fb6283d7700 1 --2- 172.21.15.153:0/3437189262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb6200f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb618009fb0 tx=0x7fb618009b30).stop 2024-01-21T20:05:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.927+0000 7fb6283d7700 1 -- 172.21.15.153:0/3437189262 shutdown_connections 2024-01-21T20:05:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.927+0000 7fb6283d7700 1 --2- 172.21.15.153:0/3437189262 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb6200f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:32.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.927+0000 7fb6283d7700 1 -- 172.21.15.153:0/3437189262 >> 172.21.15.153:0/3437189262 conn(0x7fb6200f1c70 msgr2=0x7fb6200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb6283d7700 1 -- 172.21.15.153:0/3437189262 shutdown_connections 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb6283d7700 1 -- 172.21.15.153:0/3437189262 wait complete. 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb6283d7700 1 Processor -- start 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb6283d7700 1 -- start start 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb6283d7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb620106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb6283d7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb620106c20 con 0x7fb6200f6510 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb626173700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb620106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb626173700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb620106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:47884/0 (socket says 172.21.15.153:47884) 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.928+0000 7fb626173700 1 -- 172.21.15.153:0/3728884577 learned_addr learned my addr 172.21.15.153:0/3728884577 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb626173700 1 -- 172.21.15.153:0/3728884577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb618005040 con 0x7fb6200f6510 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb626173700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb620106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb618013510 tx=0x7fb61800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb618009320 con 0x7fb6200f6510 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb618013f00 con 0x7fb6200f6510 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb620106e20 con 0x7fb6200f6510 2024-01-21T20:05:32.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb618014d90 con 0x7fb6200f6510 2024-01-21T20:05:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.929+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6201072b0 con 0x7fb6200f6510 2024-01-21T20:05:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.930+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb618014420 con 0x7fb6200f6510 2024-01-21T20:05:32.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.930+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb620045ad0 con 0x7fb6200f6510 2024-01-21T20:05:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.930+0000 7fb61effd700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb60802ff60 0x7fb608032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.930+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6180410c0 con 0x7fb6200f6510 2024-01-21T20:05:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.930+0000 7fb625972700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb60802ff60 0x7fb608032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:32.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.930+0000 7fb625972700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb60802ff60 0x7fb608032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb6140096a0 tx=0x7fb614006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:32.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:32.934+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6180095f0 con 0x7fb6200f6510 2024-01-21T20:05:33.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:32.705735+0000 mgr.smithi119.rqacbm (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:33.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.140+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6200fbf60 con 0x7fb6200f6510 2024-01-21T20:05:33.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.141+0000 7fb61effd700 1 -- 172.21.15.153:0/3728884577 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb61801db70 con 0x7fb6200f6510 2024-01-21T20:05:33.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:33.131 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:33.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb60802ff60 msgr2=0x7fb608032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb60802ff60 0x7fb608032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb6140096a0 tx=0x7fb614006b40).stop 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 msgr2=0x7fb620106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb620106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb618013510 tx=0x7fb61800b040).stop 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 shutdown_connections 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb60802ff60 0x7fb608032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 --2- 172.21.15.153:0/3728884577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb6200f6510 0x7fb620106730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.142+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 >> 172.21.15.153:0/3728884577 conn(0x7fb6200f1c70 msgr2=0x7fb6200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.143+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 shutdown_connections 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:33.143+0000 7fb6283d7700 1 -- 172.21.15.153:0/3728884577 wait complete. 2024-01-21T20:05:33.133 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:34.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:33.141513+0000 mon.smithi119 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.153:0/3728884577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:34.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:34.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:35.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:34.706106+0000 mgr.smithi119.rqacbm (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 -- 172.21.15.153:0/3558607546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 msgr2=0x7f2cdc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 --2- 172.21.15.153:0/3558607546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc0f90f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2ccc009fb0 tx=0x7f2ccc009b30).stop 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 -- 172.21.15.153:0/3558607546 shutdown_connections 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 --2- 172.21.15.153:0/3558607546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc0f90f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 -- 172.21.15.153:0/3558607546 >> 172.21.15.153:0/3558607546 conn(0x7f2cdc0f0b70 msgr2=0x7f2cdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 -- 172.21.15.153:0/3558607546 shutdown_connections 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 -- 172.21.15.153:0/3558607546 wait complete. 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 Processor -- start 2024-01-21T20:05:35.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.751+0000 7f2ce13bc700 1 -- start start 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2ce13bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2ce13bc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc106bd0 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cdaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cdaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40640/0 (socket says 172.21.15.153:40640) 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cdaffd700 1 -- 172.21.15.153:0/678956911 learned_addr learned my addr 172.21.15.153:0/678956911 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cdaffd700 1 -- 172.21.15.153:0/678956911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc005040 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cdaffd700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc1066e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2ccc013510 tx=0x7f2ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ccc009320 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ccc013f00 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc106dd0 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ccc014dc0 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.752+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc107260 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.753+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ccc014420 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.753+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc045ad0 con 0x7f2cdc060280 2024-01-21T20:05:35.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.753+0000 7f2cc37fe700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2cc402ffb0 0x7f2cc4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:35.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.753+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ccc0410b0 con 0x7f2cdc060280 2024-01-21T20:05:35.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.754+0000 7f2cda7fc700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2cc402ffb0 0x7f2cc4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:35.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.754+0000 7f2cda7fc700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2cc402ffb0 0x7f2cc4032470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2cd00096a0 tx=0x7f2cd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:35.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.758+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ccc06d2b0 con 0x7f2cdc060280 2024-01-21T20:05:35.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.960+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cdc108110 con 0x7f2cdc060280 2024-01-21T20:05:35.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.961+0000 7f2cc37fe700 1 -- 172.21.15.153:0/678956911 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2ccc0095f0 con 0x7f2cdc060280 2024-01-21T20:05:35.952 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:35.952 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:35.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.962+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2cc402ffb0 msgr2=0x7f2cc4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:35.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.962+0000 7f2ce13bc700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2cc402ffb0 0x7f2cc4032470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2cd00096a0 tx=0x7f2cd0006b40).stop 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 msgr2=0x7f2cdc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc1066e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2ccc013510 tx=0x7f2ccc00b040).stop 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 shutdown_connections 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2cc402ffb0 0x7f2cc4032470 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 --2- 172.21.15.153:0/678956911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cdc060280 0x7f2cdc1066e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 >> 172.21.15.153:0/678956911 conn(0x7f2cdc0f0b70 msgr2=0x7f2cdc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 shutdown_connections 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:35.963+0000 7f2ce13bc700 1 -- 172.21.15.153:0/678956911 wait complete. 2024-01-21T20:05:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:36.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:36 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:35.962226+0000 mon.smithi119 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.153:0/678956911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:37.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:37.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:36.706516+0000 mgr.smithi119.rqacbm (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- 172.21.15.153:0/1942823796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 msgr2=0x7f35f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 --2- 172.21.15.153:0/1942823796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f00f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f35e4009fb0 tx=0x7f35e4009b30).stop 2024-01-21T20:05:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- 172.21.15.153:0/1942823796 shutdown_connections 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 --2- 172.21.15.153:0/1942823796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f00f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- 172.21.15.153:0/1942823796 >> 172.21.15.153:0/1942823796 conn(0x7f35f00f0b70 msgr2=0x7f35f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- 172.21.15.153:0/1942823796 shutdown_connections 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- 172.21.15.153:0/1942823796 wait complete. 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 Processor -- start 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- start start 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.561+0000 7f35f6eb2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0106bc0 con 0x7f35f00f5410 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f4c4e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f4c4e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40650/0 (socket says 172.21.15.153:40650) 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f4c4e700 1 -- 172.21.15.153:0/167378197 learned_addr learned my addr 172.21.15.153:0/167378197 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f4c4e700 1 -- 172.21.15.153:0/167378197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e4005040 con 0x7f35f00f5410 2024-01-21T20:05:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f4c4e700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f01066d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f35e4013510 tx=0x7f35e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:38.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4009320 con 0x7f35f00f5410 2024-01-21T20:05:38.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f0106dc0 con 0x7f35f00f5410 2024-01-21T20:05:38.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35e4013f00 con 0x7f35f00f5410 2024-01-21T20:05:38.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35e4014d90 con 0x7f35f00f5410 2024-01-21T20:05:38.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.562+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f0107250 con 0x7f35f00f5410 2024-01-21T20:05:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.563+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35e4014420 con 0x7f35f00f5410 2024-01-21T20:05:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.563+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0045ad0 con 0x7f35f00f5410 2024-01-21T20:05:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.563+0000 7f35ed7fa700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35e002ffb0 0x7f35e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.563+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35e40410c0 con 0x7f35f00f5410 2024-01-21T20:05:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.563+0000 7f35effff700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35e002ffb0 0x7f35e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:38.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.564+0000 7f35effff700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35e002ffb0 0x7f35e0032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f35dc0096a0 tx=0x7f35dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:38.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.568+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35e40095f0 con 0x7f35f00f5410 2024-01-21T20:05:38.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.771+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35f00fae60 con 0x7f35f00f5410 2024-01-21T20:05:38.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.772+0000 7f35ed7fa700 1 -- 172.21.15.153:0/167378197 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f35e401db70 con 0x7f35f00f5410 2024-01-21T20:05:38.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:38.763 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35e002ffb0 msgr2=0x7f35e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35e002ffb0 0x7f35e0032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f35dc0096a0 tx=0x7f35dc006b40).stop 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 msgr2=0x7f35f01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f01066d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f35e4013510 tx=0x7f35e400b040).stop 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 shutdown_connections 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35e002ffb0 0x7f35e0032470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 --2- 172.21.15.153:0/167378197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f00f5410 0x7f35f01066d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 >> 172.21.15.153:0/167378197 conn(0x7f35f00f0b70 msgr2=0x7f35f005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 shutdown_connections 2024-01-21T20:05:38.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:38.774+0000 7f35f6eb2700 1 -- 172.21.15.153:0/167378197 wait complete. 2024-01-21T20:05:38.766 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:39.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:38.706797+0000 mgr.smithi119.rqacbm (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:39.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:38.773572+0000 mon.smithi119 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.153:0/167378197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:40.163 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:40.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:41.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:40.707134+0000 mgr.smithi119.rqacbm (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:41.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/363879672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 msgr2=0x7fe7e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:41.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 --2- 172.21.15.153:0/363879672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e40f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fe7d8009fb0 tx=0x7fe7d8009b30).stop 2024-01-21T20:05:41.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/363879672 shutdown_connections 2024-01-21T20:05:41.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 --2- 172.21.15.153:0/363879672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e40f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:41.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/363879672 >> 172.21.15.153:0/363879672 conn(0x7fe7e40f1c70 msgr2=0x7fe7e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/363879672 shutdown_connections 2024-01-21T20:05:41.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.412+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/363879672 wait complete. 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.413+0000 7fe7eb1c8700 1 Processor -- start 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.413+0000 7fe7eb1c8700 1 -- start start 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.413+0000 7fe7eb1c8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.413+0000 7fe7eb1c8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4106c20 con 0x7fe7e40f6510 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.413+0000 7fe7e8f64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e8f64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40674/0 (socket says 172.21.15.153:40674) 2024-01-21T20:05:41.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e8f64700 1 -- 172.21.15.153:0/575850059 learned_addr learned my addr 172.21.15.153:0/575850059 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e8f64700 1 -- 172.21.15.153:0/575850059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8005040 con 0x7fe7e40f6510 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e8f64700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e4106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe7d8014000 tx=0x7fe7d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7d80095f0 con 0x7fe7e40f6510 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e4106e20 con 0x7fe7e40f6510 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7d8007e70 con 0x7fe7e40f6510 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7d8014ea0 con 0x7fe7e40f6510 2024-01-21T20:05:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.414+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e41072b0 con 0x7fe7e40f6510 2024-01-21T20:05:41.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.415+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7d8025070 con 0x7fe7e40f6510 2024-01-21T20:05:41.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.415+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e4045ad0 con 0x7fe7e40f6510 2024-01-21T20:05:41.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.415+0000 7fe7e17fa700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe7d402ff60 0x7fe7d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:41.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.415+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7d8040940 con 0x7fe7e40f6510 2024-01-21T20:05:41.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.415+0000 7fe7e3fff700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe7d402ff60 0x7fe7d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:41.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.416+0000 7fe7e3fff700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe7d402ff60 0x7fe7d4032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe7d00096a0 tx=0x7fe7d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:41.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.420+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7d80079c0 con 0x7fe7e40f6510 2024-01-21T20:05:41.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.628+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7e40fbf60 con 0x7fe7e40f6510 2024-01-21T20:05:41.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.629+0000 7fe7e17fa700 1 -- 172.21.15.153:0/575850059 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe7d800f300 con 0x7fe7e40f6510 2024-01-21T20:05:41.620 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:41.620 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe7d402ff60 msgr2=0x7fe7d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe7d402ff60 0x7fe7d4032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe7d00096a0 tx=0x7fe7d0006b40).stop 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 msgr2=0x7fe7e4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e4106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe7d8014000 tx=0x7fe7d800b040).stop 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 shutdown_connections 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe7d402ff60 0x7fe7d4032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 --2- 172.21.15.153:0/575850059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e40f6510 0x7fe7e4106730 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 >> 172.21.15.153:0/575850059 conn(0x7fe7e40f1c70 msgr2=0x7fe7e40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 shutdown_connections 2024-01-21T20:05:41.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:41.631+0000 7fe7eb1c8700 1 -- 172.21.15.153:0/575850059 wait complete. 2024-01-21T20:05:41.623 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:42.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:41.630448+0000 mon.smithi119 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.153:0/575850059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:43.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:43.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:43.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:42.707505+0000 mgr.smithi119.rqacbm (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:44.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/561671816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 msgr2=0x7f8cdc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 --2- 172.21.15.153:0/561671816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc0f4be0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8cc4009fb0 tx=0x7f8cc4009b30).stop 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/561671816 shutdown_connections 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 --2- 172.21.15.153:0/561671816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc0f4be0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/561671816 >> 172.21.15.153:0/561671816 conn(0x7f8cdc0f02e0 msgr2=0x7f8cdc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/561671816 shutdown_connections 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.212+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/561671816 wait complete. 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8ce0ec8700 1 Processor -- start 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8ce0ec8700 1 -- start start 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8ce0ec8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8ce0ec8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc106bf0 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8cda59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8cda59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:50078/0 (socket says 172.21.15.153:50078) 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8cda59c700 1 -- 172.21.15.153:0/2598087982 learned_addr learned my addr 172.21.15.153:0/2598087982 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8cda59c700 1 -- 172.21.15.153:0/2598087982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4005040 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.213+0000 7f8cda59c700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc106700 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8cc4013510 tx=0x7f8cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:44.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc4009320 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc106df0 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc4013f00 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc4014d90 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc107280 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cc40095f0 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.214+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc045ad0 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.215+0000 7f8cd2ffd700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:44.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.215+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cc40410c0 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.215+0000 7f8cd9d9b700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:44.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.215+0000 7f8cd9d9b700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f8ccc0096a0 tx=0x7f8ccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:44.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.219+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cc4014420 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.425+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cdc108130 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.426+0000 7f8cd2ffd700 1 -- 172.21.15.153:0/2598087982 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8cc4010230 con 0x7f8cdc0f47c0 2024-01-21T20:05:44.417 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:44.417 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:44.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 msgr2=0x7f8cc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:44.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f8ccc0096a0 tx=0x7f8ccc006b40).stop 2024-01-21T20:05:44.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 msgr2=0x7f8cdc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:44.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc106700 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8cc4013510 tx=0x7f8cc400b040).stop 2024-01-21T20:05:44.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 shutdown_connections 2024-01-21T20:05:44.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:44.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 --2- 172.21.15.153:0/2598087982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8cdc0f47c0 0x7f8cdc106700 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:44.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 >> 172.21.15.153:0/2598087982 conn(0x7f8cdc0f02e0 msgr2=0x7f8cdc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:44.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 shutdown_connections 2024-01-21T20:05:44.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:44.427+0000 7f8ce0ec8700 1 -- 172.21.15.153:0/2598087982 wait complete. 2024-01-21T20:05:44.419 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:44.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:44.427400+0000 mon.smithi119 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.153:0/2598087982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:45.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:45.810 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:45.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:45 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:44.707844+0000 mgr.smithi119.rqacbm (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:46.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 -- 172.21.15.153:0/3934711529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 msgr2=0x7fa5740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 --2- 172.21.15.153:0/3934711529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fa568009fb0 tx=0x7fa568009b30).stop 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 -- 172.21.15.153:0/3934711529 shutdown_connections 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 --2- 172.21.15.153:0/3934711529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 -- 172.21.15.153:0/3934711529 >> 172.21.15.153:0/3934711529 conn(0x7fa5740f0b70 msgr2=0x7fa5740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 -- 172.21.15.153:0/3934711529 shutdown_connections 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.978+0000 7fa57b836700 1 -- 172.21.15.153:0/3934711529 wait complete. 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.979+0000 7fa57b836700 1 Processor -- start 2024-01-21T20:05:46.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.979+0000 7fa57b836700 1 -- start start 2024-01-21T20:05:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.979+0000 7fa57b836700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.979+0000 7fa57b836700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740fb820 con 0x7fa5740f5410 2024-01-21T20:05:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.979+0000 7fa5795d2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:46.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa5795d2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:50100/0 (socket says 172.21.15.153:50100) 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa5795d2700 1 -- 172.21.15.153:0/650469112 learned_addr learned my addr 172.21.15.153:0/650469112 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa5795d2700 1 -- 172.21.15.153:0/650469112 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa568005040 con 0x7fa5740f5410 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa5795d2700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740fd3a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fa568013510 tx=0x7fa56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa568009320 con 0x7fa5740f5410 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5740fb9c0 con 0x7fa5740f5410 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa568013f00 con 0x7fa5740f5410 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa568014d90 con 0x7fa5740f5410 2024-01-21T20:05:46.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.980+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5740fbe50 con 0x7fa5740f5410 2024-01-21T20:05:46.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.981+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa568014420 con 0x7fa5740f5410 2024-01-21T20:05:46.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.981+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa574059100 con 0x7fa5740f5410 2024-01-21T20:05:46.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.981+0000 7fa565ffb700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa56002ff60 0x7fa560032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:46.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.981+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5680410c0 con 0x7fa5740f5410 2024-01-21T20:05:46.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.982+0000 7fa578dd1700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa56002ff60 0x7fa560032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:46.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.982+0000 7fa578dd1700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa56002ff60 0x7fa560032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa5700096a0 tx=0x7fa570006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:46.986+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5680095f0 con 0x7fa5740f5410 2024-01-21T20:05:47.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.192+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5740fae60 con 0x7fa5740f5410 2024-01-21T20:05:47.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.192+0000 7fa565ffb700 1 -- 172.21.15.153:0/650469112 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa56801db70 con 0x7fa5740f5410 2024-01-21T20:05:47.183 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:47.183 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:47.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa56002ff60 msgr2=0x7fa560032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:47.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa56002ff60 0x7fa560032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa5700096a0 tx=0x7fa570006b40).stop 2024-01-21T20:05:47.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 msgr2=0x7fa5740fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:47.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740fd3a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fa568013510 tx=0x7fa56800b040).stop 2024-01-21T20:05:47.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 shutdown_connections 2024-01-21T20:05:47.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa56002ff60 0x7fa560032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:47.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 --2- 172.21.15.153:0/650469112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5740f5410 0x7fa5740fd3a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:47.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 >> 172.21.15.153:0/650469112 conn(0x7fa5740f0b70 msgr2=0x7fa5740f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:47.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 shutdown_connections 2024-01-21T20:05:47.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:47.194+0000 7fa57b836700 1 -- 172.21.15.153:0/650469112 wait complete. 2024-01-21T20:05:47.186 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:46.708129+0000 mgr.smithi119.rqacbm (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:48.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:48.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:48.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:47.194076+0000 mon.smithi119 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.153:0/650469112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:49.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:48.708373+0000 mgr.smithi119.rqacbm (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:49.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 -- 172.21.15.153:0/2957294606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 msgr2=0x7f90380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:49.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 --2- 172.21.15.153:0/2957294606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f90380f55c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f902c009fb0 tx=0x7f902c009b30).stop 2024-01-21T20:05:49.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 -- 172.21.15.153:0/2957294606 shutdown_connections 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 --2- 172.21.15.153:0/2957294606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f90380f55c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 -- 172.21.15.153:0/2957294606 >> 172.21.15.153:0/2957294606 conn(0x7f90380f09a0 msgr2=0x7f90380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 -- 172.21.15.153:0/2957294606 shutdown_connections 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 -- 172.21.15.153:0/2957294606 wait complete. 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 Processor -- start 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.770+0000 7f903dc33700 1 -- start start 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903dc33700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f903810b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903dc33700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9038106fc0 con 0x7f90380f51a0 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903cc31700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f903810b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903cc31700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f903810b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:50108/0 (socket says 172.21.15.153:50108) 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903cc31700 1 -- 172.21.15.153:0/554445944 learned_addr learned my addr 172.21.15.153:0/554445944 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:49.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903cc31700 1 -- 172.21.15.153:0/554445944 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c005040 con 0x7f90380f51a0 2024-01-21T20:05:49.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903cc31700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f903810b5d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f902c013510 tx=0x7f902c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:49.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902c009320 con 0x7f90380f51a0 2024-01-21T20:05:49.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9038060220 con 0x7f90380f51a0 2024-01-21T20:05:49.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f902c013f00 con 0x7f90380f51a0 2024-01-21T20:05:49.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902c014dc0 con 0x7f90380f51a0 2024-01-21T20:05:49.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.771+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903810bac0 con 0x7f90380f51a0 2024-01-21T20:05:49.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.772+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f902c0263a0 con 0x7f90380f51a0 2024-01-21T20:05:49.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.772+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9038045ad0 con 0x7f90380f51a0 2024-01-21T20:05:49.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.772+0000 7f90357fa700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f902802ffb0 0x7f9028032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:49.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.772+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f902c041330 con 0x7f90380f51a0 2024-01-21T20:05:49.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.773+0000 7f9037fff700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f902802ffb0 0x7f9028032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:49.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.773+0000 7f9037fff700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f902802ffb0 0x7f9028032470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f90240096a0 tx=0x7f9024006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:49.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.777+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f902c01cb10 con 0x7f90380f51a0 2024-01-21T20:05:49.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.982+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f903810c280 con 0x7f90380f51a0 2024-01-21T20:05:49.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.982+0000 7f90357fa700 1 -- 172.21.15.153:0/554445944 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f902c00f340 con 0x7f90380f51a0 2024-01-21T20:05:49.973 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:49.973 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f902802ffb0 msgr2=0x7f9028032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f902802ffb0 0x7f9028032470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f90240096a0 tx=0x7f9024006b40).stop 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 msgr2=0x7f903810b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f903810b5d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f902c013510 tx=0x7f902c00b040).stop 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 shutdown_connections 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f902802ffb0 0x7f9028032470 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 --2- 172.21.15.153:0/554445944 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90380f51a0 0x7f903810b5d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:49.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 >> 172.21.15.153:0/554445944 conn(0x7f90380f09a0 msgr2=0x7f90380f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:49.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 shutdown_connections 2024-01-21T20:05:49.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:49.984+0000 7f903dc33700 1 -- 172.21.15.153:0/554445944 wait complete. 2024-01-21T20:05:49.976 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:50.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:49.983786+0000 mon.smithi119 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.153:0/554445944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:51.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:51.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:51.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:50.708712+0000 mgr.smithi119.rqacbm (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 -- 172.21.15.153:0/4149279449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 msgr2=0x7faf400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 --2- 172.21.15.153:0/4149279449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf400f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7faf30009fb0 tx=0x7faf30009b30).stop 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 -- 172.21.15.153:0/4149279449 shutdown_connections 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 --2- 172.21.15.153:0/4149279449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf400f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 -- 172.21.15.153:0/4149279449 >> 172.21.15.153:0/4149279449 conn(0x7faf400f1c70 msgr2=0x7faf400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 -- 172.21.15.153:0/4149279449 shutdown_connections 2024-01-21T20:05:52.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.574+0000 7faf481d1700 1 -- 172.21.15.153:0/4149279449 wait complete. 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf481d1700 1 Processor -- start 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf481d1700 1 -- start start 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf481d1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf4006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf481d1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf40069e20 con 0x7faf400f6510 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf45f6d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf4006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf45f6d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf4006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:50132/0 (socket says 172.21.15.153:50132) 2024-01-21T20:05:52.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.575+0000 7faf45f6d700 1 -- 172.21.15.153:0/1298787611 learned_addr learned my addr 172.21.15.153:0/1298787611 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:52.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf45f6d700 1 -- 172.21.15.153:0/1298787611 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30005040 con 0x7faf400f6510 2024-01-21T20:05:52.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf45f6d700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf4006b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7faf30013510 tx=0x7faf3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:52.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf30009320 con 0x7faf400f6510 2024-01-21T20:05:52.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf40069fc0 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf30013f00 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf30014d90 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.576+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4006a450 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.577+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf30014420 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.577+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf40059100 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.577+0000 7faf367fc700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faf2c02ff60 0x7faf2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.577+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf300410d0 con 0x7faf400f6510 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.577+0000 7faf4576c700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faf2c02ff60 0x7faf2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.577+0000 7faf4576c700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faf2c02ff60 0x7faf2c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7faf3c0096a0 tx=0x7faf3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:52.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.581+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf300095f0 con 0x7faf400f6510 2024-01-21T20:05:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.787+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf400fbf60 con 0x7faf400f6510 2024-01-21T20:05:52.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.788+0000 7faf367fc700 1 -- 172.21.15.153:0/1298787611 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf3001db70 con 0x7faf400f6510 2024-01-21T20:05:52.779 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:52.779 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faf2c02ff60 msgr2=0x7faf2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faf2c02ff60 0x7faf2c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7faf3c0096a0 tx=0x7faf3c006b40).stop 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 msgr2=0x7faf4006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf4006b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7faf30013510 tx=0x7faf3000b040).stop 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 shutdown_connections 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faf2c02ff60 0x7faf2c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 --2- 172.21.15.153:0/1298787611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faf400f6510 0x7faf4006b9a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 >> 172.21.15.153:0/1298787611 conn(0x7faf400f1c70 msgr2=0x7faf400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 shutdown_connections 2024-01-21T20:05:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:52.790+0000 7faf481d1700 1 -- 172.21.15.153:0/1298787611 wait complete. 2024-01-21T20:05:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:53.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:52.709069+0000 mgr.smithi119.rqacbm (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:53.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:52.789513+0000 mon.smithi119 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.153:0/1298787611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:54.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:54.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:55.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:54.709419+0000 mgr.smithi119.rqacbm (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 -- 172.21.15.153:0/717903995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 msgr2=0x7fb3900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 --2- 172.21.15.153:0/717903995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3900f90f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fb378009fb0 tx=0x7fb378009b30).stop 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 -- 172.21.15.153:0/717903995 shutdown_connections 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 --2- 172.21.15.153:0/717903995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3900f90f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 -- 172.21.15.153:0/717903995 >> 172.21.15.153:0/717903995 conn(0x7fb3900f0b70 msgr2=0x7fb3900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 -- 172.21.15.153:0/717903995 shutdown_connections 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.340+0000 7fb396972700 1 -- 172.21.15.153:0/717903995 wait complete. 2024-01-21T20:05:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb396972700 1 Processor -- start 2024-01-21T20:05:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb396972700 1 -- start start 2024-01-21T20:05:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb396972700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb396972700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb390106bd0 con 0x7fb390060280 2024-01-21T20:05:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb38ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44306/0 (socket says 172.21.15.153:44306) 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.341+0000 7fb38ffff700 1 -- 172.21.15.153:0/2486440907 learned_addr learned my addr 172.21.15.153:0/2486440907 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb38ffff700 1 -- 172.21.15.153:0/2486440907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378005040 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb38ffff700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3901066e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fb378013510 tx=0x7fb37800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb378009320 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb390106dd0 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb378013f00 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb378014dc0 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.342+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb390107260 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.343+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3780095f0 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.343+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb390045ad0 con 0x7fb390060280 2024-01-21T20:05:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.343+0000 7fb38cff9700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb37c02ff60 0x7fb37c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.343+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3780410e0 con 0x7fb390060280 2024-01-21T20:05:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.343+0000 7fb38f7fe700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb37c02ff60 0x7fb37c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.344+0000 7fb38f7fe700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb37c02ff60 0x7fb37c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb3800096a0 tx=0x7fb380006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.348+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb378014420 con 0x7fb390060280 2024-01-21T20:05:55.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.555+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3901080b0 con 0x7fb390060280 2024-01-21T20:05:55.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.556+0000 7fb38cff9700 1 -- 172.21.15.153:0/2486440907 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb37801c920 con 0x7fb390060280 2024-01-21T20:05:55.547 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:55.547 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb37c02ff60 msgr2=0x7fb37c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb37c02ff60 0x7fb37c032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb3800096a0 tx=0x7fb380006b40).stop 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 msgr2=0x7fb3901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3901066e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fb378013510 tx=0x7fb37800b040).stop 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 shutdown_connections 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb37c02ff60 0x7fb37c032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 --2- 172.21.15.153:0/2486440907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb390060280 0x7fb3901066e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 >> 172.21.15.153:0/2486440907 conn(0x7fb3900f0b70 msgr2=0x7fb3900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 shutdown_connections 2024-01-21T20:05:55.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:55.558+0000 7fb396972700 1 -- 172.21.15.153:0/2486440907 wait complete. 2024-01-21T20:05:55.550 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:56.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:55 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:55.557718+0000 mon.smithi119 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.153:0/2486440907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:56.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:56.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:56.709776+0000 mgr.smithi119.rqacbm (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 -- 172.21.15.153:0/495453691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 msgr2=0x7f8ce00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 --2- 172.21.15.153:0/495453691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce00f55c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8cd0009fb0 tx=0x7f8cd0009b30).stop 2024-01-21T20:05:58.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 -- 172.21.15.153:0/495453691 shutdown_connections 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 --2- 172.21.15.153:0/495453691 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce00f55c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 -- 172.21.15.153:0/495453691 >> 172.21.15.153:0/495453691 conn(0x7f8ce00f09a0 msgr2=0x7f8ce00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 -- 172.21.15.153:0/495453691 shutdown_connections 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 -- 172.21.15.153:0/495453691 wait complete. 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.135+0000 7f8ce5125700 1 Processor -- start 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8ce5125700 1 -- start start 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8ce5125700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce01064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8ce5125700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce01069b0 con 0x7f8ce00f51a0 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8cdffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce01064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8cdffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce01064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44330/0 (socket says 172.21.15.153:44330) 2024-01-21T20:05:58.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8cdffff700 1 -- 172.21.15.153:0/139947470 learned_addr learned my addr 172.21.15.153:0/139947470 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8cdffff700 1 -- 172.21.15.153:0/139947470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0005040 con 0x7f8ce00f51a0 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8cdffff700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce01064c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f8cd0013510 tx=0x7f8cd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd0009320 con 0x7f8ce00f51a0 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.136+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce0106bb0 con 0x7f8ce00f51a0 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cd0013f00 con 0x7f8ce00f51a0 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd0014dc0 con 0x7f8ce00f51a0 2024-01-21T20:05:58.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce0107040 con 0x7f8ce00f51a0 2024-01-21T20:05:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cd00095f0 con 0x7f8ce00f51a0 2024-01-21T20:05:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce0045ad0 con 0x7f8ce00f51a0 2024-01-21T20:05:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8cdcff9700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:05:58.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.137+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cd0041000 con 0x7f8ce00f51a0 2024-01-21T20:05:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.138+0000 7f8cdf7fe700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:05:58.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.138+0000 7f8cdf7fe700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8cd40096a0 tx=0x7f8cd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:05:58.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.142+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cd0014420 con 0x7f8ce00f51a0 2024-01-21T20:05:58.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.346+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ce0107ef0 con 0x7f8ce00f51a0 2024-01-21T20:05:58.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.347+0000 7f8cdcff9700 1 -- 172.21.15.153:0/139947470 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8cd0010230 con 0x7f8ce00f51a0 2024-01-21T20:05:58.338 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:05:58.338 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:05:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 msgr2=0x7f8cc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f8cd40096a0 tx=0x7f8cd4006b40).stop 2024-01-21T20:05:58.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 msgr2=0x7f8ce01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce01064c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f8cd0013510 tx=0x7f8cd000b040).stop 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 shutdown_connections 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8cc802ff60 0x7f8cc8032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 --2- 172.21.15.153:0/139947470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ce00f51a0 0x7f8ce01064c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 >> 172.21.15.153:0/139947470 conn(0x7f8ce00f09a0 msgr2=0x7f8ce00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 shutdown_connections 2024-01-21T20:05:58.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:05:58.348+0000 7f8ce5125700 1 -- 172.21.15.153:0/139947470 wait complete. 2024-01-21T20:05:58.341 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:05:58.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:58.348338+0000 mon.smithi119 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.153:0/139947470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:05:59.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:05:59.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:05:59.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:05:58.710163+0000 mgr.smithi119.rqacbm (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:05:59.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:58.785784+0000 mon.smithi119 (mon.0) 253 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:05:59.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:05:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:05:58.786022+0000 mon.smithi119 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:06:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 -- 172.21.15.153:0/463532941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 msgr2=0x7f6f4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 --2- 172.21.15.153:0/463532941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f6f3c009fb0 tx=0x7f6f3c009b30).stop 2024-01-21T20:06:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 -- 172.21.15.153:0/463532941 shutdown_connections 2024-01-21T20:06:00.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 --2- 172.21.15.153:0/463532941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c0f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 -- 172.21.15.153:0/463532941 >> 172.21.15.153:0/463532941 conn(0x7f6f4c0f0b70 msgr2=0x7f6f4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 -- 172.21.15.153:0/463532941 shutdown_connections 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.952+0000 7f6f54658700 1 -- 172.21.15.153:0/463532941 wait complete. 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.953+0000 7f6f54658700 1 Processor -- start 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.953+0000 7f6f54658700 1 -- start start 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.953+0000 7f6f54658700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.953+0000 7f6f54658700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c106bc0 con 0x7f6f4c0f5410 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f523f4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f523f4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:44350/0 (socket says 172.21.15.153:44350) 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f523f4700 1 -- 172.21.15.153:0/927430401 learned_addr learned my addr 172.21.15.153:0/927430401 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f523f4700 1 -- 172.21.15.153:0/927430401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c005040 con 0x7f6f4c0f5410 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f523f4700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c1066d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6f3c013510 tx=0x7f6f3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:00.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f3c009320 con 0x7f6f4c0f5410 2024-01-21T20:06:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f3c013f00 con 0x7f6f4c0f5410 2024-01-21T20:06:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c106dc0 con 0x7f6f4c0f5410 2024-01-21T20:06:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f3c014d90 con 0x7f6f4c0f5410 2024-01-21T20:06:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.954+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c107250 con 0x7f6f4c0f5410 2024-01-21T20:06:00.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.955+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f3c0095f0 con 0x7f6f4c0f5410 2024-01-21T20:06:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.955+0000 7f6f42ffd700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f3802ff60 0x7f6f38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.955+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c045ad0 con 0x7f6f4c0f5410 2024-01-21T20:06:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.955+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f3c041000 con 0x7f6f4c0f5410 2024-01-21T20:06:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.955+0000 7f6f51bf3700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f3802ff60 0x7f6f38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:00.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.956+0000 7f6f51bf3700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f3802ff60 0x7f6f38032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6f480096a0 tx=0x7f6f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:00.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:00.960+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f3c06b2b0 con 0x7f6f4c0f5410 2024-01-21T20:06:01.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:00.710563+0000 mgr.smithi119.rqacbm (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:01.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.167+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f4c108100 con 0x7f6f4c0f5410 2024-01-21T20:06:01.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.167+0000 7f6f42ffd700 1 -- 172.21.15.153:0/927430401 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f3c01d7f0 con 0x7f6f4c0f5410 2024-01-21T20:06:01.158 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:01.158 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f3802ff60 msgr2=0x7f6f38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f3802ff60 0x7f6f38032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6f480096a0 tx=0x7f6f48006b40).stop 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 msgr2=0x7f6f4c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c1066d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f6f3c013510 tx=0x7f6f3c00b040).stop 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 shutdown_connections 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f6f3802ff60 0x7f6f38032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 --2- 172.21.15.153:0/927430401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f4c0f5410 0x7f6f4c1066d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 >> 172.21.15.153:0/927430401 conn(0x7f6f4c0f0b70 msgr2=0x7f6f4c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:01.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 shutdown_connections 2024-01-21T20:06:01.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:01.169+0000 7f6f54658700 1 -- 172.21.15.153:0/927430401 wait complete. 2024-01-21T20:06:01.161 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:02.555 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:02.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:02.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:01.168763+0000 mon.smithi119 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.153:0/927430401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:03.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:02.710938+0000 mgr.smithi119.rqacbm (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:03.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 -- 172.21.15.153:0/2524435427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 msgr2=0x7f66dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:03.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 --2- 172.21.15.153:0/2524435427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc0f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f66cc009fb0 tx=0x7f66cc009b30).stop 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 -- 172.21.15.153:0/2524435427 shutdown_connections 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 --2- 172.21.15.153:0/2524435427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc0f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 -- 172.21.15.153:0/2524435427 >> 172.21.15.153:0/2524435427 conn(0x7f66dc0f0b70 msgr2=0x7f66dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 -- 172.21.15.153:0/2524435427 shutdown_connections 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 -- 172.21.15.153:0/2524435427 wait complete. 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 Processor -- start 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.727+0000 7f66e10ba700 1 -- start start 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66e10ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66e10ba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc105b20 con 0x7f66dc0f5410 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66dad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66dad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34418/0 (socket says 172.21.15.153:34418) 2024-01-21T20:06:03.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66dad9d700 1 -- 172.21.15.153:0/1337534558 learned_addr learned my addr 172.21.15.153:0/1337534558 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66dad9d700 1 -- 172.21.15.153:0/1337534558 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66cc005040 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66dad9d700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f66cc013510 tx=0x7f66cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66cc009320 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66cc013f00 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc105d20 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66cc014d90 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.728+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc1061b0 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.729+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66cc0263a0 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.729+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc045ad0 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.729+0000 7f66c37fe700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.729+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66cc040fe0 con 0x7f66dc0f5410 2024-01-21T20:06:03.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.730+0000 7f66da59c700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:03.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.730+0000 7f66da59c700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f66c402ff60 0x7f66c4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f66d00096a0 tx=0x7f66d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:03.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.734+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66cc0126f0 con 0x7f66dc0f5410 2024-01-21T20:06:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.941+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66dc0fae60 con 0x7f66dc0f5410 2024-01-21T20:06:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.941+0000 7f66c37fe700 1 -- 172.21.15.153:0/1337534558 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66cc00f340 con 0x7f66dc0f5410 2024-01-21T20:06:03.933 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:03.933 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f66c402ff60 msgr2=0x7f66c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f66c402ff60 0x7f66c4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f66d00096a0 tx=0x7f66d0006b40).stop 2024-01-21T20:06:03.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 msgr2=0x7f66dc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:03.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f66cc013510 tx=0x7f66cc00b040).stop 2024-01-21T20:06:03.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 shutdown_connections 2024-01-21T20:06:03.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:03.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 --2- 172.21.15.153:0/1337534558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66dc0f5410 0x7f66dc105630 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:03.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 >> 172.21.15.153:0/1337534558 conn(0x7f66dc0f0b70 msgr2=0x7f66dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:03.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 shutdown_connections 2024-01-21T20:06:03.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:03.943+0000 7f66e10ba700 1 -- 172.21.15.153:0/1337534558 wait complete. 2024-01-21T20:06:03.936 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:04.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:03.942907+0000 mon.smithi119 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.153:0/1337534558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:05.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:05.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:05.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:04.711198+0000 mgr.smithi119.rqacbm (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.457+0000 7f12fffff700 1 -- 172.21.15.153:0/3092982143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 msgr2=0x7f13000f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 --2- 172.21.15.153:0/3092982143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f13000f8f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f12f0009fb0 tx=0x7f12f0009b30).stop 2024-01-21T20:06:06.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 -- 172.21.15.153:0/3092982143 shutdown_connections 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 --2- 172.21.15.153:0/3092982143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f13000f8f20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 -- 172.21.15.153:0/3092982143 >> 172.21.15.153:0/3092982143 conn(0x7f13000f09a0 msgr2=0x7f13000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 -- 172.21.15.153:0/3092982143 shutdown_connections 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 -- 172.21.15.153:0/3092982143 wait complete. 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 Processor -- start 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.458+0000 7f12fffff700 1 -- start start 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12fffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f1300106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12fffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13001069f0 con 0x7f13000601e0 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12feffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f1300106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12feffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f1300106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34434/0 (socket says 172.21.15.153:34434) 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12feffd700 1 -- 172.21.15.153:0/3088475281 learned_addr learned my addr 172.21.15.153:0/3088475281 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12feffd700 1 -- 172.21.15.153:0/3088475281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0005040 con 0x7f13000601e0 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12feffd700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f1300106500 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f12f0013510 tx=0x7f12f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12f0009320 con 0x7f13000601e0 2024-01-21T20:06:06.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1300106bf0 con 0x7f13000601e0 2024-01-21T20:06:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12f0013f00 con 0x7f13000601e0 2024-01-21T20:06:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12f0014d90 con 0x7f13000601e0 2024-01-21T20:06:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.459+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1300107080 con 0x7f13000601e0 2024-01-21T20:06:06.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.460+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12f0014420 con 0x7f13000601e0 2024-01-21T20:06:06.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.460+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1300045ad0 con 0x7f13000601e0 2024-01-21T20:06:06.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.460+0000 7f12e77fe700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f12e802ff60 0x7f12e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:06.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.460+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12f0041070 con 0x7f13000601e0 2024-01-21T20:06:06.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.461+0000 7f12fe7fc700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f12e802ff60 0x7f12e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:06.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.461+0000 7f12fe7fc700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f12e802ff60 0x7f12e8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f12f40096a0 tx=0x7f12f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:06.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.465+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12f00095f0 con 0x7f13000601e0 2024-01-21T20:06:06.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.674+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1300107f30 con 0x7f13000601e0 2024-01-21T20:06:06.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.674+0000 7f12e77fe700 1 -- 172.21.15.153:0/3088475281 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12f001db70 con 0x7f13000601e0 2024-01-21T20:06:06.665 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:06.665 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f12e802ff60 msgr2=0x7f12e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f12e802ff60 0x7f12e8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f12f40096a0 tx=0x7f12f4006b40).stop 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 msgr2=0x7f1300106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f1300106500 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f12f0013510 tx=0x7f12f000b040).stop 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 shutdown_connections 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f12e802ff60 0x7f12e8032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 --2- 172.21.15.153:0/3088475281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13000601e0 0x7f1300106500 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 >> 172.21.15.153:0/3088475281 conn(0x7f13000f09a0 msgr2=0x7f13000f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 shutdown_connections 2024-01-21T20:06:06.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:06.676+0000 7f12fffff700 1 -- 172.21.15.153:0/3088475281 wait complete. 2024-01-21T20:06:06.668 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:06.675854+0000 mon.smithi119 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.153:0/3088475281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:06.711526+0000 mgr.smithi119.rqacbm (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:07.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:07.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:09.089 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:08.711879+0000 mgr.smithi119.rqacbm (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 -- 172.21.15.153:0/2565799775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 msgr2=0x7f3b080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 --2- 172.21.15.153:0/2565799775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b080f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f3af8009fb0 tx=0x7f3af8009b30).stop 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 -- 172.21.15.153:0/2565799775 shutdown_connections 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 --2- 172.21.15.153:0/2565799775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b080f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 -- 172.21.15.153:0/2565799775 >> 172.21.15.153:0/2565799775 conn(0x7f3b080f1c70 msgr2=0x7f3b080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 -- 172.21.15.153:0/2565799775 shutdown_connections 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.200+0000 7f3b0f772700 1 -- 172.21.15.153:0/2565799775 wait complete. 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0f772700 1 Processor -- start 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0f772700 1 -- start start 2024-01-21T20:06:09.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0f772700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b08106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0f772700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08106c20 con 0x7f3b080f6510 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0d50e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b08106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0d50e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b08106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34454/0 (socket says 172.21.15.153:34454) 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.201+0000 7f3b0d50e700 1 -- 172.21.15.153:0/784175599 learned_addr learned my addr 172.21.15.153:0/784175599 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.202+0000 7f3b0d50e700 1 -- 172.21.15.153:0/784175599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8005040 con 0x7f3b080f6510 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.202+0000 7f3b0d50e700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b08106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3af8013510 tx=0x7f3af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.202+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af8009320 con 0x7f3b080f6510 2024-01-21T20:06:09.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.202+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3af8013f00 con 0x7f3b080f6510 2024-01-21T20:06:09.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.202+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b08106e20 con 0x7f3b080f6510 2024-01-21T20:06:09.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.202+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3af8014d90 con 0x7f3b080f6510 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.203+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b081072b0 con 0x7f3b080f6510 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.203+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3af8014420 con 0x7f3b080f6510 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.203+0000 7f3afdffb700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3af402ff60 0x7f3af4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.203+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3af8045940 con 0x7f3b080f6510 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.204+0000 7f3b0cd0d700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3af402ff60 0x7f3af4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.204+0000 7f3b0cd0d700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3af402ff60 0x7f3af4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3b040096a0 tx=0x7f3b04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:09.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.205+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b08059100 con 0x7f3b080f6510 2024-01-21T20:06:09.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.209+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3af800e2a0 con 0x7f3b080f6510 2024-01-21T20:06:09.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.414+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b080fbf60 con 0x7f3b080f6510 2024-01-21T20:06:09.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.414+0000 7f3afdffb700 1 -- 172.21.15.153:0/784175599 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3af801bac0 con 0x7f3b080f6510 2024-01-21T20:06:09.405 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:09.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3af402ff60 msgr2=0x7f3af4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3af402ff60 0x7f3af4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3b040096a0 tx=0x7f3b04006b40).stop 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 msgr2=0x7f3b08106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b08106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f3af8013510 tx=0x7f3af800b040).stop 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 shutdown_connections 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3af402ff60 0x7f3af4032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 --2- 172.21.15.153:0/784175599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b080f6510 0x7f3b08106730 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 >> 172.21.15.153:0/784175599 conn(0x7f3b080f1c70 msgr2=0x7f3b080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 shutdown_connections 2024-01-21T20:06:09.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:09.416+0000 7f3b0f772700 1 -- 172.21.15.153:0/784175599 wait complete. 2024-01-21T20:06:09.408 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:10.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:09.415783+0000 mon.smithi119 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.153:0/784175599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:10.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:10.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:11.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:10.712163+0000 mgr.smithi119.rqacbm (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:12.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.086+0000 7fbfd359e700 1 -- 172.21.15.153:0/115078882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 msgr2=0x7fbfd40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:12.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.086+0000 7fbfd359e700 1 --2- 172.21.15.153:0/115078882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd40f66c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fbfc4009fb0 tx=0x7fbfc4009b30).stop 2024-01-21T20:06:12.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.086+0000 7fbfd359e700 1 -- 172.21.15.153:0/115078882 shutdown_connections 2024-01-21T20:06:12.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.086+0000 7fbfd359e700 1 --2- 172.21.15.153:0/115078882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd40f66c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:12.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.086+0000 7fbfd359e700 1 -- 172.21.15.153:0/115078882 >> 172.21.15.153:0/115078882 conn(0x7fbfd40f1aa0 msgr2=0x7fbfd40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:12.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.086+0000 7fbfd359e700 1 -- 172.21.15.153:0/115078882 shutdown_connections 2024-01-21T20:06:12.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.087+0000 7fbfd359e700 1 -- 172.21.15.153:0/115078882 wait complete. 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.087+0000 7fbfd359e700 1 Processor -- start 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.087+0000 7fbfd359e700 1 -- start start 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.087+0000 7fbfd359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.087+0000 7fbfd359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd4069e20 con 0x7fbfd40f62a0 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34476/0 (socket says 172.21.15.153:34476) 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd259c700 1 -- 172.21.15.153:0/430019792 learned_addr learned my addr 172.21.15.153:0/430019792 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:12.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd259c700 1 -- 172.21.15.153:0/430019792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc4005040 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd259c700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd406b9a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fbfc4013510 tx=0x7fbfc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc4009320 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd4069fc0 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfc4013f00 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc4014dc0 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.088+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd406a450 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.089+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfc40095f0 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.089+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd4059100 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.089+0000 7fbfcaffd700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbfc002ff60 0x7fbfc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.089+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfc4041040 con 0x7fbfd40f62a0 2024-01-21T20:06:12.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.089+0000 7fbfd1d9b700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbfc002ff60 0x7fbfc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:12.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.090+0000 7fbfd1d9b700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbfc002ff60 0x7fbfc0032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fbfbc0096a0 tx=0x7fbfbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:12.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.094+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfc4014420 con 0x7fbfd40f62a0 2024-01-21T20:06:12.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.299+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfd40fbce0 con 0x7fbfd40f62a0 2024-01-21T20:06:12.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.299+0000 7fbfcaffd700 1 -- 172.21.15.153:0/430019792 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbfc4010230 con 0x7fbfd40f62a0 2024-01-21T20:06:12.290 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:12.290 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbfc002ff60 msgr2=0x7fbfc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbfc002ff60 0x7fbfc0032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fbfbc0096a0 tx=0x7fbfbc006b40).stop 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 msgr2=0x7fbfd406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd406b9a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fbfc4013510 tx=0x7fbfc400b040).stop 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 shutdown_connections 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fbfc002ff60 0x7fbfc0032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 --2- 172.21.15.153:0/430019792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbfd40f62a0 0x7fbfd406b9a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 >> 172.21.15.153:0/430019792 conn(0x7fbfd40f1aa0 msgr2=0x7fbfd40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:12.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 shutdown_connections 2024-01-21T20:06:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:12.301+0000 7fbfd359e700 1 -- 172.21.15.153:0/430019792 wait complete. 2024-01-21T20:06:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:12.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:12.300884+0000 mon.smithi119 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.153:0/430019792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:13.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:13.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:13.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:12.712450+0000 mgr.smithi119.rqacbm (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:14.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 -- 172.21.15.153:0/4087584522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 msgr2=0x7f0e340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 --2- 172.21.15.153:0/4087584522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e340f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f0e28009fb0 tx=0x7f0e28009b30).stop 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 -- 172.21.15.153:0/4087584522 shutdown_connections 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 --2- 172.21.15.153:0/4087584522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e340f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 -- 172.21.15.153:0/4087584522 >> 172.21.15.153:0/4087584522 conn(0x7f0e340f0b70 msgr2=0x7f0e340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 -- 172.21.15.153:0/4087584522 shutdown_connections 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.938+0000 7f0e39259700 1 -- 172.21.15.153:0/4087584522 wait complete. 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e39259700 1 Processor -- start 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e39259700 1 -- start start 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e39259700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e34106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:14.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e39259700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e34106c10 con 0x7f0e340f5410 2024-01-21T20:06:14.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e32d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e34106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e32d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e34106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:35898/0 (socket says 172.21.15.153:35898) 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.939+0000 7f0e32d9d700 1 -- 172.21.15.153:0/3174893351 learned_addr learned my addr 172.21.15.153:0/3174893351 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e32d9d700 1 -- 172.21.15.153:0/3174893351 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28005040 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e32d9d700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e34106720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0e28013510 tx=0x7f0e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28009320 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e28013f00 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e34106e10 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e28014d90 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.940+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e341072a0 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.941+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e34045ad0 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.941+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e280095f0 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.941+0000 7f0e1b7fe700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0e1c02ff60 0x7f0e1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.941+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e28040fe0 con 0x7f0e340f5410 2024-01-21T20:06:14.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.941+0000 7f0e3259c700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0e1c02ff60 0x7f0e1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:14.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.941+0000 7f0e3259c700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0e1c02ff60 0x7f0e1c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f0e240096a0 tx=0x7f0e24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:14.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:14.946+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e28014420 con 0x7f0e340f5410 2024-01-21T20:06:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:14.712756+0000 mgr.smithi119.rqacbm (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:15.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.148+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e341080f0 con 0x7f0e340f5410 2024-01-21T20:06:15.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.148+0000 7f0e1b7fe700 1 -- 172.21.15.153:0/3174893351 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0e28010230 con 0x7f0e340f5410 2024-01-21T20:06:15.139 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:15.139 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0e1c02ff60 msgr2=0x7f0e1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0e1c02ff60 0x7f0e1c032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f0e240096a0 tx=0x7f0e24006b40).stop 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 msgr2=0x7f0e34106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e34106720 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0e28013510 tx=0x7f0e2800b040).stop 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 shutdown_connections 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0e1c02ff60 0x7f0e1c032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 --2- 172.21.15.153:0/3174893351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e340f5410 0x7f0e34106720 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 >> 172.21.15.153:0/3174893351 conn(0x7f0e340f0b70 msgr2=0x7f0e3405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 shutdown_connections 2024-01-21T20:06:15.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:15.150+0000 7f0e39259700 1 -- 172.21.15.153:0/3174893351 wait complete. 2024-01-21T20:06:15.142 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:16.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:15 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:15.149892+0000 mon.smithi119 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.153:0/3174893351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:16.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:16.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:16.713037+0000 mgr.smithi119.rqacbm (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- 172.21.15.153:0/211563649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 msgr2=0x7fb3140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 --2- 172.21.15.153:0/211563649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb3140f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb304009fb0 tx=0x7fb304009b30).stop 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- 172.21.15.153:0/211563649 shutdown_connections 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 --2- 172.21.15.153:0/211563649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb3140f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- 172.21.15.153:0/211563649 >> 172.21.15.153:0/211563649 conn(0x7fb3140f0b70 msgr2=0x7fb3140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- 172.21.15.153:0/211563649 shutdown_connections 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- 172.21.15.153:0/211563649 wait complete. 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 Processor -- start 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- start start 2024-01-21T20:06:17.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb314109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb31b80a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb314060240 con 0x7fb3140f5410 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.706+0000 7fb3195a6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb314109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb3195a6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb314109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:35916/0 (socket says 172.21.15.153:35916) 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb3195a6700 1 -- 172.21.15.153:0/3460344992 learned_addr learned my addr 172.21.15.153:0/3460344992 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb3195a6700 1 -- 172.21.15.153:0/3460344992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb304005040 con 0x7fb3140f5410 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb3195a6700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb314109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb304013510 tx=0x7fb30400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb304009320 con 0x7fb3140f5410 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb314060440 con 0x7fb3140f5410 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb304013f00 con 0x7fb3140f5410 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb304014d90 con 0x7fb3140f5410 2024-01-21T20:06:17.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.707+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb314109ae0 con 0x7fb3140f5410 2024-01-21T20:06:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.708+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3040263a0 con 0x7fb3140f5410 2024-01-21T20:06:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.708+0000 7fb309ffb700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb30002ffb0 0x7fb300032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.708+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb314045ad0 con 0x7fb3140f5410 2024-01-21T20:06:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.708+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb304041390 con 0x7fb3140f5410 2024-01-21T20:06:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.709+0000 7fb318da5700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb30002ffb0 0x7fb300032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:17.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.709+0000 7fb318da5700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb30002ffb0 0x7fb300032470 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb3100096a0 tx=0x7fb310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:17.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.714+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb304012920 con 0x7fb3140f5410 2024-01-21T20:06:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.917+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3140faf50 con 0x7fb3140f5410 2024-01-21T20:06:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.917+0000 7fb309ffb700 1 -- 172.21.15.153:0/3460344992 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb304012330 con 0x7fb3140f5410 2024-01-21T20:06:17.909 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:17.909 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb30002ffb0 msgr2=0x7fb300032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb30002ffb0 0x7fb300032470 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb3100096a0 tx=0x7fb310006b40).stop 2024-01-21T20:06:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 msgr2=0x7fb314109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb314109430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb304013510 tx=0x7fb30400b040).stop 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 shutdown_connections 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb30002ffb0 0x7fb300032470 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 --2- 172.21.15.153:0/3460344992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3140f5410 0x7fb314109430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 >> 172.21.15.153:0/3460344992 conn(0x7fb3140f0b70 msgr2=0x7fb3140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 shutdown_connections 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:17.919+0000 7fb31b80a700 1 -- 172.21.15.153:0/3460344992 wait complete. 2024-01-21T20:06:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:18.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:17.918915+0000 mon.smithi119 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.153:0/3460344992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:19.330 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:19.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:19.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:18.713272+0000 mgr.smithi119.rqacbm (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:20.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.545+0000 7f7d814b2700 1 -- 172.21.15.153:0/3218306550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 msgr2=0x7f7d7c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:20.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.545+0000 7f7d814b2700 1 --2- 172.21.15.153:0/3218306550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c0f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f7d6c009fb0 tx=0x7f7d6c009b30).stop 2024-01-21T20:06:20.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 -- 172.21.15.153:0/3218306550 shutdown_connections 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 --2- 172.21.15.153:0/3218306550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c0f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 -- 172.21.15.153:0/3218306550 >> 172.21.15.153:0/3218306550 conn(0x7f7d7c0f1c70 msgr2=0x7f7d7c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 -- 172.21.15.153:0/3218306550 shutdown_connections 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 -- 172.21.15.153:0/3218306550 wait complete. 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 Processor -- start 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.546+0000 7f7d814b2700 1 -- start start 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d814b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d814b2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7c069e20 con 0x7f7d7c0f6510 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d7affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d7affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:35932/0 (socket says 172.21.15.153:35932) 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d7affd700 1 -- 172.21.15.153:0/2793996455 learned_addr learned my addr 172.21.15.153:0/2793996455 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d7affd700 1 -- 172.21.15.153:0/2793996455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c005040 con 0x7f7d7c0f6510 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d7affd700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c06b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7d6c013510 tx=0x7f7d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:20.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d6c009320 con 0x7f7d7c0f6510 2024-01-21T20:06:20.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d6c013f00 con 0x7f7d7c0f6510 2024-01-21T20:06:20.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7c069fc0 con 0x7f7d7c0f6510 2024-01-21T20:06:20.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d6c014d90 con 0x7f7d7c0f6510 2024-01-21T20:06:20.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.547+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7c06a450 con 0x7f7d7c0f6510 2024-01-21T20:06:20.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.548+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d6c0263a0 con 0x7f7d7c0f6510 2024-01-21T20:06:20.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.548+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7c059100 con 0x7f7d7c0f6510 2024-01-21T20:06:20.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.548+0000 7f7d637fe700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d6402ffb0 0x7f7d64032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:20.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.548+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d6c040ff0 con 0x7f7d7c0f6510 2024-01-21T20:06:20.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.549+0000 7f7d7a7fc700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d6402ffb0 0x7f7d64032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:20.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.549+0000 7f7d7a7fc700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d6402ffb0 0x7f7d64032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:20.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.553+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d6c06d2b0 con 0x7f7d7c0f6510 2024-01-21T20:06:20.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.758+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d7c0fbf60 con 0x7f7d7c0f6510 2024-01-21T20:06:20.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.759+0000 7f7d637fe700 1 -- 172.21.15.153:0/2793996455 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d6c0095f0 con 0x7f7d7c0f6510 2024-01-21T20:06:20.749 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:20.749 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d6402ffb0 msgr2=0x7f7d64032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d6402ffb0 0x7f7d64032470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f7d700096a0 tx=0x7f7d70006b40).stop 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 msgr2=0x7f7d7c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c06b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f7d6c013510 tx=0x7f7d6c00b040).stop 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 shutdown_connections 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f7d6402ffb0 0x7f7d64032470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 --2- 172.21.15.153:0/2793996455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d7c0f6510 0x7f7d7c06b9a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 >> 172.21.15.153:0/2793996455 conn(0x7f7d7c0f1c70 msgr2=0x7f7d7c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 shutdown_connections 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:20.761+0000 7f7d814b2700 1 -- 172.21.15.153:0/2793996455 wait complete. 2024-01-21T20:06:20.751 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:21.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:20.713678+0000 mgr.smithi119.rqacbm (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:21.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:20.759601+0000 mon.smithi119 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.153:0/2793996455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:22.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:22.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:23.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:22.714041+0000 mgr.smithi119.rqacbm (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- 172.21.15.153:0/1859900222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 msgr2=0x7f63280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 --2- 172.21.15.153:0/1859900222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63280f55c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f6318009fb0 tx=0x7f6318009b30).stop 2024-01-21T20:06:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- 172.21.15.153:0/1859900222 shutdown_connections 2024-01-21T20:06:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 --2- 172.21.15.153:0/1859900222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63280f55c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- 172.21.15.153:0/1859900222 >> 172.21.15.153:0/1859900222 conn(0x7f63280f09a0 msgr2=0x7f63280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- 172.21.15.153:0/1859900222 shutdown_connections 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- 172.21.15.153:0/1859900222 wait complete. 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 Processor -- start 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- start start 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63281064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.340+0000 7f632e8c7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63281069d0 con 0x7f63280f51a0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632d8c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63281064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632d8c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63281064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:45440/0 (socket says 172.21.15.153:45440) 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632d8c5700 1 -- 172.21.15.153:0/1743574689 learned_addr learned my addr 172.21.15.153:0/1743574689 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632d8c5700 1 -- 172.21.15.153:0/1743574689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318005040 con 0x7f63280f51a0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632d8c5700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63281064e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f6318013510 tx=0x7f631800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6318009320 con 0x7f63280f51a0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328106bd0 con 0x7f63280f51a0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6318013f00 con 0x7f63280f51a0 2024-01-21T20:06:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6318014dc0 con 0x7f63280f51a0 2024-01-21T20:06:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.341+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6328107060 con 0x7f63280f51a0 2024-01-21T20:06:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.342+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63180095f0 con 0x7f63280f51a0 2024-01-21T20:06:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.342+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6328045ad0 con 0x7f63280f51a0 2024-01-21T20:06:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.342+0000 7f631e7fc700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f631402ff60 0x7f6314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.342+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6318041100 con 0x7f63280f51a0 2024-01-21T20:06:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.342+0000 7f632d0c4700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f631402ff60 0x7f6314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.343+0000 7f632d0c4700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f631402ff60 0x7f6314032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f63240096a0 tx=0x7f6324006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.347+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6318014420 con 0x7f63280f51a0 2024-01-21T20:06:23.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.554+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6328107f10 con 0x7f63280f51a0 2024-01-21T20:06:23.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.555+0000 7f631e7fc700 1 -- 172.21.15.153:0/1743574689 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6318010230 con 0x7f63280f51a0 2024-01-21T20:06:23.545 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:23.545 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:23.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f631402ff60 msgr2=0x7f6314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:23.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f631402ff60 0x7f6314032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f63240096a0 tx=0x7f6324006b40).stop 2024-01-21T20:06:23.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 msgr2=0x7f63281064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:23.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63281064e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f6318013510 tx=0x7f631800b040).stop 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 shutdown_connections 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f631402ff60 0x7f6314032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 --2- 172.21.15.153:0/1743574689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63280f51a0 0x7f63281064e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 >> 172.21.15.153:0/1743574689 conn(0x7f63280f09a0 msgr2=0x7f632805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 shutdown_connections 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:23.556+0000 7f632e8c7700 1 -- 172.21.15.153:0/1743574689 wait complete. 2024-01-21T20:06:23.547 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:24.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:23.555011+0000 mon.smithi119 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.153:0/1743574689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:24.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:24.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:25.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:24.714353+0000 mgr.smithi119.rqacbm (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:26.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 -- 172.21.15.153:0/1159006915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 msgr2=0x7fb0d00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 --2- 172.21.15.153:0/1159006915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d00f66c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fb0c0009fb0 tx=0x7fb0c0009b30).stop 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 -- 172.21.15.153:0/1159006915 shutdown_connections 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 --2- 172.21.15.153:0/1159006915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d00f66c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 -- 172.21.15.153:0/1159006915 >> 172.21.15.153:0/1159006915 conn(0x7fb0d00f1aa0 msgr2=0x7fb0d00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 -- 172.21.15.153:0/1159006915 shutdown_connections 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 -- 172.21.15.153:0/1159006915 wait complete. 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.091+0000 7fb0d4820700 1 Processor -- start 2024-01-21T20:06:26.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.092+0000 7fb0d4820700 1 -- start start 2024-01-21T20:06:26.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.092+0000 7fb0d4820700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:26.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.092+0000 7fb0d4820700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d01069a0 con 0x7fb0d00f62a0 2024-01-21T20:06:26.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.092+0000 7fb0cf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:26.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.092+0000 7fb0cf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:45466/0 (socket says 172.21.15.153:45466) 2024-01-21T20:06:26.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.092+0000 7fb0cf7fe700 1 -- 172.21.15.153:0/3257260614 learned_addr learned my addr 172.21.15.153:0/3257260614 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:26.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0cf7fe700 1 -- 172.21.15.153:0/3257260614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c0005040 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0cf7fe700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d01064b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb0c0013510 tx=0x7fb0c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0c0009320 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d0106ba0 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0c0013f00 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0c0014dc0 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.093+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d0107030 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.094+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0c0014420 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.094+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d0045ad0 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.094+0000 7fb0b7fff700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb0b802ff60 0x7fb0b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.094+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0c0041050 con 0x7fb0d00f62a0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.094+0000 7fb0ceffd700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb0b802ff60 0x7fb0b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:26.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.095+0000 7fb0ceffd700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb0b802ff60 0x7fb0b8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb0c40096a0 tx=0x7fb0c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:26.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.099+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0c00095f0 con 0x7fb0d00f62a0 2024-01-21T20:06:26.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.307+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0d0107ee0 con 0x7fb0d00f62a0 2024-01-21T20:06:26.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.308+0000 7fb0b7fff700 1 -- 172.21.15.153:0/3257260614 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb0c001cb10 con 0x7fb0d00f62a0 2024-01-21T20:06:26.297 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:26.298 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:26.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb0b802ff60 msgr2=0x7fb0b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:26.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb0b802ff60 0x7fb0b8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fb0c40096a0 tx=0x7fb0c4006b40).stop 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 msgr2=0x7fb0d01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d01064b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb0c0013510 tx=0x7fb0c000b040).stop 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 shutdown_connections 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb0b802ff60 0x7fb0b8032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 --2- 172.21.15.153:0/3257260614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0d00f62a0 0x7fb0d01064b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 >> 172.21.15.153:0/3257260614 conn(0x7fb0d00f1aa0 msgr2=0x7fb0d00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 shutdown_connections 2024-01-21T20:06:26.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:26.309+0000 7fb0d4820700 1 -- 172.21.15.153:0/3257260614 wait complete. 2024-01-21T20:06:26.303 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:26.308123+0000 mon.smithi119 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.153:0/3257260614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:27.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:26.714665+0000 mgr.smithi119.rqacbm (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:27.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:27.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 -- 172.21.15.153:0/2913409675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 msgr2=0x7fd7b00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 --2- 172.21.15.153:0/2913409675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f90f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fd7a4009fb0 tx=0x7fd7a4009b30).stop 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 -- 172.21.15.153:0/2913409675 shutdown_connections 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 --2- 172.21.15.153:0/2913409675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f90f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 -- 172.21.15.153:0/2913409675 >> 172.21.15.153:0/2913409675 conn(0x7fd7b00f0b70 msgr2=0x7fd7b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 -- 172.21.15.153:0/2913409675 shutdown_connections 2024-01-21T20:06:28.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.923+0000 7fd7b502a700 1 -- 172.21.15.153:0/2913409675 wait complete. 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7b502a700 1 Processor -- start 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7b502a700 1 -- start start 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7b502a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7b502a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b00f4ff0 con 0x7fd7b0060280 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7aed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7aed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:45474/0 (socket says 172.21.15.153:45474) 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7aed9d700 1 -- 172.21.15.153:0/256966743 learned_addr learned my addr 172.21.15.153:0/256966743 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7aed9d700 1 -- 172.21.15.153:0/256966743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4005040 con 0x7fd7b0060280 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.924+0000 7fd7aed9d700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f6b70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fd7a4013510 tx=0x7fd7a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a4009320 con 0x7fd7b0060280 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b00f5190 con 0x7fd7b0060280 2024-01-21T20:06:28.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7a4013f00 con 0x7fd7b0060280 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a4014dc0 con 0x7fd7b0060280 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b00f5620 con 0x7fd7b0060280 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7a40095f0 con 0x7fd7b0060280 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.925+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b0059100 con 0x7fd7b0060280 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.926+0000 7fd7977fe700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd79802ff60 0x7fd798032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.926+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7a4041000 con 0x7fd7b0060280 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.926+0000 7fd7ae59c700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd79802ff60 0x7fd798032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:28.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.926+0000 7fd7ae59c700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd79802ff60 0x7fd798032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd7a00096a0 tx=0x7fd7a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:28.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:28.930+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7a4014420 con 0x7fd7b0060280 2024-01-21T20:06:29.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:28.714912+0000 mgr.smithi119.rqacbm (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:29.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.134+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7b00fbf50 con 0x7fd7b0060280 2024-01-21T20:06:29.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.135+0000 7fd7977fe700 1 -- 172.21.15.153:0/256966743 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd7a4010230 con 0x7fd7b0060280 2024-01-21T20:06:29.125 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:29.125 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:29.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd79802ff60 msgr2=0x7fd798032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:29.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd79802ff60 0x7fd798032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fd7a00096a0 tx=0x7fd7a0006b40).stop 2024-01-21T20:06:29.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 msgr2=0x7fd7b00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f6b70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fd7a4013510 tx=0x7fd7a400b040).stop 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 shutdown_connections 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd79802ff60 0x7fd798032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 --2- 172.21.15.153:0/256966743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7b0060280 0x7fd7b00f6b70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 >> 172.21.15.153:0/256966743 conn(0x7fd7b00f0b70 msgr2=0x7fd7b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 shutdown_connections 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:29.137+0000 7fd7b502a700 1 -- 172.21.15.153:0/256966743 wait complete. 2024-01-21T20:06:29.127 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:30.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:29.135590+0000 mon.smithi119 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.153:0/256966743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:30.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:30.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:31.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:30.715233+0000 mgr.smithi119.rqacbm (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 -- 172.21.15.153:0/248099326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 msgr2=0x7f33dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 --2- 172.21.15.153:0/248099326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc0f55c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f33cc009fb0 tx=0x7f33cc009b30).stop 2024-01-21T20:06:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 -- 172.21.15.153:0/248099326 shutdown_connections 2024-01-21T20:06:31.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 --2- 172.21.15.153:0/248099326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc0f55c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 -- 172.21.15.153:0/248099326 >> 172.21.15.153:0/248099326 conn(0x7f33dc0f09a0 msgr2=0x7f33dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 -- 172.21.15.153:0/248099326 shutdown_connections 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 -- 172.21.15.153:0/248099326 wait complete. 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 Processor -- start 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.679+0000 7f33e2f7a700 1 -- start start 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e2f7a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e2f7a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc060240 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e1f78700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e1f78700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:45498/0 (socket says 172.21.15.153:45498) 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e1f78700 1 -- 172.21.15.153:0/3489593617 learned_addr learned my addr 172.21.15.153:0/3489593617 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e1f78700 1 -- 172.21.15.153:0/3489593617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc005040 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.680+0000 7f33e1f78700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc1091b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f33cc013510 tx=0x7f33cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.681+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc009320 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.681+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc060440 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.681+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc109860 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.681+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33cc013f00 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.681+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc014d90 con 0x7f33dc0f51a0 2024-01-21T20:06:31.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.682+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33cc0095f0 con 0x7f33dc0f51a0 2024-01-21T20:06:31.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.681+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc045ad0 con 0x7f33dc0f51a0 2024-01-21T20:06:31.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.682+0000 7f33d27fc700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f33c802ff60 0x7f33c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:31.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.682+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33cc041020 con 0x7f33dc0f51a0 2024-01-21T20:06:31.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.682+0000 7f33e1777700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f33c802ff60 0x7f33c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:31.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.682+0000 7f33e1777700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f33c802ff60 0x7f33c8032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f33d80096a0 tx=0x7f33d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.687+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33cc014420 con 0x7f33dc0f51a0 2024-01-21T20:06:31.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.894+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33dc10a080 con 0x7f33dc0f51a0 2024-01-21T20:06:31.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.895+0000 7f33d27fc700 1 -- 172.21.15.153:0/3489593617 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f33cc012980 con 0x7f33dc0f51a0 2024-01-21T20:06:31.886 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:31.886 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f33c802ff60 msgr2=0x7f33c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f33c802ff60 0x7f33c8032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f33d80096a0 tx=0x7f33d8006b40).stop 2024-01-21T20:06:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 msgr2=0x7f33dc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc1091b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f33cc013510 tx=0x7f33cc00b040).stop 2024-01-21T20:06:31.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 shutdown_connections 2024-01-21T20:06:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f33c802ff60 0x7f33c8032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 --2- 172.21.15.153:0/3489593617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33dc0f51a0 0x7f33dc1091b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 >> 172.21.15.153:0/3489593617 conn(0x7f33dc0f09a0 msgr2=0x7f33dc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 shutdown_connections 2024-01-21T20:06:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:31.896+0000 7f33e2f7a700 1 -- 172.21.15.153:0/3489593617 wait complete. 2024-01-21T20:06:31.888 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:32.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:31.896339+0000 mon.smithi119 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.153:0/3489593617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:33.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:33.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:33.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:32.715643+0000 mgr.smithi119.rqacbm (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.473+0000 7fa57866d700 1 -- 172.21.15.153:0/2873398815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 msgr2=0x7fa5700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.473+0000 7fa57866d700 1 --2- 172.21.15.153:0/2873398815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa5700f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fa560009fb0 tx=0x7fa560009b30).stop 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 -- 172.21.15.153:0/2873398815 shutdown_connections 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 --2- 172.21.15.153:0/2873398815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa5700f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 -- 172.21.15.153:0/2873398815 >> 172.21.15.153:0/2873398815 conn(0x7fa5700f0b70 msgr2=0x7fa5700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 -- 172.21.15.153:0/2873398815 shutdown_connections 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 -- 172.21.15.153:0/2873398815 wait complete. 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 Processor -- start 2024-01-21T20:06:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 -- start start 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa570105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.474+0000 7fa57866d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa570105af0 con 0x7fa5700f5410 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa576409700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa570105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa576409700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa570105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40042/0 (socket says 172.21.15.153:40042) 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa576409700 1 -- 172.21.15.153:0/2391576130 learned_addr learned my addr 172.21.15.153:0/2391576130 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa576409700 1 -- 172.21.15.153:0/2391576130 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560005040 con 0x7fa5700f5410 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa576409700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa570105600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa560013510 tx=0x7fa56000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa560009320 con 0x7fa5700f5410 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570105cf0 con 0x7fa5700f5410 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa560013f00 con 0x7fa5700f5410 2024-01-21T20:06:34.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa560014d90 con 0x7fa5700f5410 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.475+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa570106180 con 0x7fa5700f5410 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.476+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5600095f0 con 0x7fa5700f5410 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.476+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa570045ad0 con 0x7fa5700f5410 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.476+0000 7fa566ffd700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa55c02ff60 0x7fa55c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.476+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa560041000 con 0x7fa5700f5410 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.476+0000 7fa575c08700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa55c02ff60 0x7fa55c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:34.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.477+0000 7fa575c08700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa55c02ff60 0x7fa55c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa56c0096a0 tx=0x7fa56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:34.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.481+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa560014420 con 0x7fa5700f5410 2024-01-21T20:06:34.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.688+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5700fae60 con 0x7fa5700f5410 2024-01-21T20:06:34.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.688+0000 7fa566ffd700 1 -- 172.21.15.153:0/2391576130 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa560010230 con 0x7fa5700f5410 2024-01-21T20:06:34.679 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:34.680 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa55c02ff60 msgr2=0x7fa55c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa55c02ff60 0x7fa55c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa56c0096a0 tx=0x7fa56c006b40).stop 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 msgr2=0x7fa570105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa570105600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fa560013510 tx=0x7fa56000b040).stop 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 shutdown_connections 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa55c02ff60 0x7fa55c032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:34.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 --2- 172.21.15.153:0/2391576130 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5700f5410 0x7fa570105600 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:34.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 >> 172.21.15.153:0/2391576130 conn(0x7fa5700f0b70 msgr2=0x7fa5700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:34.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 shutdown_connections 2024-01-21T20:06:34.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:34.690+0000 7fa57866d700 1 -- 172.21.15.153:0/2391576130 wait complete. 2024-01-21T20:06:34.682 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:35.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:34.689750+0000 mon.smithi119 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.153:0/2391576130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:36.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:34.715980+0000 mgr.smithi119.rqacbm (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:36.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:36.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:37.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4048666848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 msgr2=0x7f8be00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:37.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 --2- 172.21.15.153:0/4048666848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be00f6930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f8bd0009fb0 tx=0x7f8bd0009b30).stop 2024-01-21T20:06:37.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4048666848 shutdown_connections 2024-01-21T20:06:37.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 --2- 172.21.15.153:0/4048666848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be00f6930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:37.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4048666848 >> 172.21.15.153:0/4048666848 conn(0x7f8be00f1c70 msgr2=0x7f8be00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:37.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4048666848 shutdown_connections 2024-01-21T20:06:37.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.241+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4048666848 wait complete. 2024-01-21T20:06:37.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be7ab7700 1 Processor -- start 2024-01-21T20:06:37.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be7ab7700 1 -- start start 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be7ab7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be7ab7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be0106c20 con 0x7f8be00f6510 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be5853700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be5853700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40066/0 (socket says 172.21.15.153:40066) 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be5853700 1 -- 172.21.15.153:0/4029911231 learned_addr learned my addr 172.21.15.153:0/4029911231 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be5853700 1 -- 172.21.15.153:0/4029911231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd0005040 con 0x7f8be00f6510 2024-01-21T20:06:37.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.242+0000 7f8be5853700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be0106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8bd0013510 tx=0x7f8bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.243+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bd0009320 con 0x7f8be00f6510 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.243+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be0106e20 con 0x7f8be00f6510 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.243+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bd0013f00 con 0x7f8be00f6510 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.243+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8bd0014d90 con 0x7f8be00f6510 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.243+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be01072b0 con 0x7f8be00f6510 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.243+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8bd0014420 con 0x7f8be00f6510 2024-01-21T20:06:37.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.244+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be0045ad0 con 0x7f8be00f6510 2024-01-21T20:06:37.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.244+0000 7f8bd67fc700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8bcc02ff60 0x7f8bcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:37.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.244+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8bd0040fb0 con 0x7f8be00f6510 2024-01-21T20:06:37.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.244+0000 7f8be5052700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8bcc02ff60 0x7f8bcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:37.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.244+0000 7f8be5052700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8bcc02ff60 0x7f8bcc032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8bdc0096a0 tx=0x7f8bdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:37.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.248+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8bd00095f0 con 0x7f8be00f6510 2024-01-21T20:06:37.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.455+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8be00fbf60 con 0x7f8be00f6510 2024-01-21T20:06:37.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.456+0000 7f8bd67fc700 1 -- 172.21.15.153:0/4029911231 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8bd001db70 con 0x7f8be00f6510 2024-01-21T20:06:37.447 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:37.447 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:37.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.457+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8bcc02ff60 msgr2=0x7f8bcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:37.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.457+0000 7f8be7ab7700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8bcc02ff60 0x7f8bcc032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8bdc0096a0 tx=0x7f8bdc006b40).stop 2024-01-21T20:06:37.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.457+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 msgr2=0x7f8be0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:37.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.457+0000 7f8be7ab7700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be0106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f8bd0013510 tx=0x7f8bd000b040).stop 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.458+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 shutdown_connections 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.458+0000 7f8be7ab7700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8bcc02ff60 0x7f8bcc032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.458+0000 7f8be7ab7700 1 --2- 172.21.15.153:0/4029911231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8be00f6510 0x7f8be0106730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.458+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 >> 172.21.15.153:0/4029911231 conn(0x7f8be00f1c70 msgr2=0x7f8be00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.458+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 shutdown_connections 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:37.458+0000 7f8be7ab7700 1 -- 172.21.15.153:0/4029911231 wait complete. 2024-01-21T20:06:37.450 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:36.716343+0000 mgr.smithi119.rqacbm (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:38.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:37.457401+0000 mon.smithi119 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.153:0/4029911231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:38.881 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:38.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:39.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:38.716713+0000 mgr.smithi119.rqacbm (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.061+0000 7f20b6b23700 1 -- 172.21.15.153:0/2129524102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 msgr2=0x7f20b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.061+0000 7f20b6b23700 1 --2- 172.21.15.153:0/2129524102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b00f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f20a4009fb0 tx=0x7f20a4009b30).stop 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 -- 172.21.15.153:0/2129524102 shutdown_connections 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 --2- 172.21.15.153:0/2129524102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b00f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 -- 172.21.15.153:0/2129524102 >> 172.21.15.153:0/2129524102 conn(0x7f20b00f0b70 msgr2=0x7f20b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 -- 172.21.15.153:0/2129524102 shutdown_connections 2024-01-21T20:06:40.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 -- 172.21.15.153:0/2129524102 wait complete. 2024-01-21T20:06:40.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 Processor -- start 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.062+0000 7f20b6b23700 1 -- start start 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b6b23700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b6b23700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b0060280 con 0x7f20b00f5410 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b48bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b48bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40084/0 (socket says 172.21.15.153:40084) 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b48bf700 1 -- 172.21.15.153:0/3559981531 learned_addr learned my addr 172.21.15.153:0/3559981531 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b48bf700 1 -- 172.21.15.153:0/3559981531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4005040 con 0x7f20b00f5410 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b48bf700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b01093f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f20a4013510 tx=0x7f20a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a4009320 con 0x7f20b00f5410 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20a4013f00 con 0x7f20b00f5410 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b0060480 con 0x7f20b00f5410 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.063+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20a4014d90 con 0x7f20b00f5410 2024-01-21T20:06:40.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.064+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b0109b20 con 0x7f20b00f5410 2024-01-21T20:06:40.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.064+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20a4014420 con 0x7f20b00f5410 2024-01-21T20:06:40.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.064+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b0045ad0 con 0x7f20b00f5410 2024-01-21T20:06:40.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.064+0000 7f20ad7fa700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f20a002ff60 0x7f20a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:40.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.064+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20a4041000 con 0x7f20b00f5410 2024-01-21T20:06:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.065+0000 7f20affff700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f20a002ff60 0x7f20a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:40.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.065+0000 7f20affff700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f20a002ff60 0x7f20a0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f209c0096a0 tx=0x7f209c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:40.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.069+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20a4026bf0 con 0x7f20b00f5410 2024-01-21T20:06:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.276+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20b010a340 con 0x7f20b00f5410 2024-01-21T20:06:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.276+0000 7f20ad7fa700 1 -- 172.21.15.153:0/3559981531 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f20a401db70 con 0x7f20b00f5410 2024-01-21T20:06:40.267 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:40.268 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f20a002ff60 msgr2=0x7f20a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f20a002ff60 0x7f20a0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f209c0096a0 tx=0x7f209c006b40).stop 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 msgr2=0x7f20b01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b01093f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f20a4013510 tx=0x7f20a400b040).stop 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 shutdown_connections 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f20a002ff60 0x7f20a0032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 --2- 172.21.15.153:0/3559981531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20b00f5410 0x7f20b01093f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:40.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 >> 172.21.15.153:0/3559981531 conn(0x7f20b00f0b70 msgr2=0x7f20b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 shutdown_connections 2024-01-21T20:06:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:40.278+0000 7f20b6b23700 1 -- 172.21.15.153:0/3559981531 wait complete. 2024-01-21T20:06:40.270 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:40.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:40.277888+0000 mon.smithi119 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.153:0/3559981531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:41.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:40.717045+0000 mgr.smithi119.rqacbm (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:41.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:41.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 -- 172.21.15.153:0/949089548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 msgr2=0x7f36500f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 --2- 172.21.15.153:0/949089548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36500f5810 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f3638009fb0 tx=0x7f3638009b30).stop 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 -- 172.21.15.153:0/949089548 shutdown_connections 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 --2- 172.21.15.153:0/949089548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36500f5810 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 -- 172.21.15.153:0/949089548 >> 172.21.15.153:0/949089548 conn(0x7f36500f0b70 msgr2=0x7f36500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 -- 172.21.15.153:0/949089548 shutdown_connections 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.762+0000 7f3655a9a700 1 -- 172.21.15.153:0/949089548 wait complete. 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f3655a9a700 1 Processor -- start 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f3655a9a700 1 -- start start 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f3655a9a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36501055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:42.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f3655a9a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3650105ad0 con 0x7f36500f53f0 2024-01-21T20:06:42.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f364f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36501055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:42.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f364f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36501055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40112/0 (socket says 172.21.15.153:40112) 2024-01-21T20:06:42.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.763+0000 7f364f7fe700 1 -- 172.21.15.153:0/2395386079 learned_addr learned my addr 172.21.15.153:0/2395386079 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:42.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f364f7fe700 1 -- 172.21.15.153:0/2395386079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3638005040 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f364f7fe700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36501055e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3638013510 tx=0x7f363800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3638009320 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3650105cd0 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3638013f00 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3638014d90 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.764+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3650106160 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.765+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f36380095f0 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.765+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3650045ad0 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.765+0000 7f3647fff700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f363c02ff60 0x7f363c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.765+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3638041020 con 0x7f36500f53f0 2024-01-21T20:06:42.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.766+0000 7f364effd700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f363c02ff60 0x7f363c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:42.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.766+0000 7f364effd700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f363c02ff60 0x7f363c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f36400096a0 tx=0x7f3640006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:42.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.770+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3638014420 con 0x7f36500f53f0 2024-01-21T20:06:42.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.977+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3650107010 con 0x7f36500f53f0 2024-01-21T20:06:42.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.978+0000 7f3647fff700 1 -- 172.21.15.153:0/2395386079 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f363801d7f0 con 0x7f36500f53f0 2024-01-21T20:06:42.969 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:42.969 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:42.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f363c02ff60 msgr2=0x7f363c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:42.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f363c02ff60 0x7f363c032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f36400096a0 tx=0x7f3640006b40).stop 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 msgr2=0x7f36501055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36501055e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3638013510 tx=0x7f363800b040).stop 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 shutdown_connections 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f363c02ff60 0x7f363c032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 --2- 172.21.15.153:0/2395386079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36500f53f0 0x7f36501055e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 >> 172.21.15.153:0/2395386079 conn(0x7f36500f0b70 msgr2=0x7f36500f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 shutdown_connections 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:42.980+0000 7f3655a9a700 1 -- 172.21.15.153:0/2395386079 wait complete. 2024-01-21T20:06:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:43.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:42.717413+0000 mgr.smithi119.rqacbm (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:44.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:42.979554+0000 mon.smithi119 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.153:0/2395386079' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:44.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:44.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:45.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:44.717837+0000 mgr.smithi119.rqacbm (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:45.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 -- 172.21.15.153:0/2443060309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 msgr2=0x7f25640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:45.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 --2- 172.21.15.153:0/2443060309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f25640f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f2554009fb0 tx=0x7f2554009b30).stop 2024-01-21T20:06:45.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 -- 172.21.15.153:0/2443060309 shutdown_connections 2024-01-21T20:06:45.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 --2- 172.21.15.153:0/2443060309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f25640f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:45.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 -- 172.21.15.153:0/2443060309 >> 172.21.15.153:0/2443060309 conn(0x7f25640f1c70 msgr2=0x7f25640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:45.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 -- 172.21.15.153:0/2443060309 shutdown_connections 2024-01-21T20:06:45.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 -- 172.21.15.153:0/2443060309 wait complete. 2024-01-21T20:06:45.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 Processor -- start 2024-01-21T20:06:45.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.481+0000 7f256bb08700 1 -- start start 2024-01-21T20:06:45.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f256bb08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f2564106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:45.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f256bb08700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2564106c20 con 0x7f25640f6510 2024-01-21T20:06:45.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f25698a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f2564106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f25698a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f2564106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:52540/0 (socket says 172.21.15.153:52540) 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f25698a4700 1 -- 172.21.15.153:0/3577153574 learned_addr learned my addr 172.21.15.153:0/3577153574 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f25698a4700 1 -- 172.21.15.153:0/3577153574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2554005040 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f25698a4700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f2564106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f2554014000 tx=0x7f255400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25540095f0 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2564106e20 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2554007e70 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.482+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2554014ea0 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.483+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25641072b0 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.483+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2554025070 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.483+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2564045ad0 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.483+0000 7f255a7fc700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f255002ff60 0x7f2550032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.483+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25540409f0 con 0x7f25640f6510 2024-01-21T20:06:45.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.484+0000 7f25690a3700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f255002ff60 0x7f2550032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:45.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.484+0000 7f25690a3700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f255002ff60 0x7f2550032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f25600096a0 tx=0x7f2560006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:45.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.488+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25540079c0 con 0x7f25640f6510 2024-01-21T20:06:45.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.696+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25640fbf60 con 0x7f25640f6510 2024-01-21T20:06:45.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.697+0000 7f255a7fc700 1 -- 172.21.15.153:0/3577153574 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f255400f300 con 0x7f25640f6510 2024-01-21T20:06:45.688 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:45.688 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f255002ff60 msgr2=0x7f2550032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f255002ff60 0x7f2550032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f25600096a0 tx=0x7f2560006b40).stop 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 msgr2=0x7f2564106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f2564106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f2554014000 tx=0x7f255400b040).stop 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 shutdown_connections 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f255002ff60 0x7f2550032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:45.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 --2- 172.21.15.153:0/3577153574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25640f6510 0x7f2564106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:45.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 >> 172.21.15.153:0/3577153574 conn(0x7f25640f1c70 msgr2=0x7f25640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:45.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 shutdown_connections 2024-01-21T20:06:45.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:45.698+0000 7f256bb08700 1 -- 172.21.15.153:0/3577153574 wait complete. 2024-01-21T20:06:45.690 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:46.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:45 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:45.698204+0000 mon.smithi119 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.153:0/3577153574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:47.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:47.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:46.718200+0000 mgr.smithi119.rqacbm (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 -- 172.21.15.153:0/894601071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 msgr2=0x7f44940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 --2- 172.21.15.153:0/894601071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f44940f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4484009fb0 tx=0x7f4484009b30).stop 2024-01-21T20:06:48.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 -- 172.21.15.153:0/894601071 shutdown_connections 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 --2- 172.21.15.153:0/894601071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f44940f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 -- 172.21.15.153:0/894601071 >> 172.21.15.153:0/894601071 conn(0x7f44940f1c70 msgr2=0x7f44940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 -- 172.21.15.153:0/894601071 shutdown_connections 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.235+0000 7f449c4dd700 1 -- 172.21.15.153:0/894601071 wait complete. 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449c4dd700 1 Processor -- start 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449c4dd700 1 -- start start 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449c4dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f4494106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449c4dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494106c20 con 0x7f44940f6510 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449a279700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f4494106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449a279700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f4494106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:52556/0 (socket says 172.21.15.153:52556) 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449a279700 1 -- 172.21.15.153:0/2528271697 learned_addr learned my addr 172.21.15.153:0/2528271697 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449a279700 1 -- 172.21.15.153:0/2528271697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484005040 con 0x7f44940f6510 2024-01-21T20:06:48.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f449a279700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f4494106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4484013510 tx=0x7f448400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:48.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.236+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4484009320 con 0x7f44940f6510 2024-01-21T20:06:48.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.237+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4494106e20 con 0x7f44940f6510 2024-01-21T20:06:48.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.237+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4484013f00 con 0x7f44940f6510 2024-01-21T20:06:48.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.237+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4484014d90 con 0x7f44940f6510 2024-01-21T20:06:48.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.237+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44941072b0 con 0x7f44940f6510 2024-01-21T20:06:48.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.237+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44840095f0 con 0x7f44940f6510 2024-01-21T20:06:48.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.237+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4494045ad0 con 0x7f44940f6510 2024-01-21T20:06:48.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.238+0000 7f448affd700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f448002ff60 0x7f4480032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:48.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.238+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4484041000 con 0x7f44940f6510 2024-01-21T20:06:48.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.238+0000 7f4499a78700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f448002ff60 0x7f4480032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:48.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.238+0000 7f4499a78700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f448002ff60 0x7f4480032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f44900096a0 tx=0x7f4490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:48.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.242+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4484014420 con 0x7f44940f6510 2024-01-21T20:06:48.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.450+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44940fbf60 con 0x7f44940f6510 2024-01-21T20:06:48.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.451+0000 7f448affd700 1 -- 172.21.15.153:0/2528271697 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f448401d7f0 con 0x7f44940f6510 2024-01-21T20:06:48.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:48.442 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f448002ff60 msgr2=0x7f4480032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f448002ff60 0x7f4480032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f44900096a0 tx=0x7f4490006b40).stop 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 msgr2=0x7f4494106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f4494106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4484013510 tx=0x7f448400b040).stop 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 shutdown_connections 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f448002ff60 0x7f4480032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 --2- 172.21.15.153:0/2528271697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44940f6510 0x7f4494106730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 >> 172.21.15.153:0/2528271697 conn(0x7f44940f1c70 msgr2=0x7f44940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 shutdown_connections 2024-01-21T20:06:48.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:48.452+0000 7f449c4dd700 1 -- 172.21.15.153:0/2528271697 wait complete. 2024-01-21T20:06:48.444 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:48.836 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:48.452217+0000 mon.smithi119 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.153:0/2528271697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:49.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:49.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:49.836 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:48.718609+0000 mgr.smithi119.rqacbm (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:50.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.942+0000 7fa457fff700 1 -- 172.21.15.153:0/3714197983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 msgr2=0x7fa4580f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.942+0000 7fa457fff700 1 --2- 172.21.15.153:0/3714197983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa4580f8f20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fa448009fb0 tx=0x7fa448009b30).stop 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 -- 172.21.15.153:0/3714197983 shutdown_connections 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 --2- 172.21.15.153:0/3714197983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa4580f8f20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 -- 172.21.15.153:0/3714197983 >> 172.21.15.153:0/3714197983 conn(0x7fa4580f09a0 msgr2=0x7fa4580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 -- 172.21.15.153:0/3714197983 shutdown_connections 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 -- 172.21.15.153:0/3714197983 wait complete. 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 Processor -- start 2024-01-21T20:06:50.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 -- start start 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa458106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.943+0000 7fa457fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4581069f0 con 0x7fa4580601e0 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa456ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa458106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa456ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa458106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:52576/0 (socket says 172.21.15.153:52576) 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa456ffd700 1 -- 172.21.15.153:0/1017246072 learned_addr learned my addr 172.21.15.153:0/1017246072 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa456ffd700 1 -- 172.21.15.153:0/1017246072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa448005040 con 0x7fa4580601e0 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa456ffd700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa458106500 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa448013510 tx=0x7fa44800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa448009320 con 0x7fa4580601e0 2024-01-21T20:06:50.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa458106bf0 con 0x7fa4580601e0 2024-01-21T20:06:50.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa448013f00 con 0x7fa4580601e0 2024-01-21T20:06:50.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa448014d90 con 0x7fa4580601e0 2024-01-21T20:06:50.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.944+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa458107080 con 0x7fa4580601e0 2024-01-21T20:06:50.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.945+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa458045ad0 con 0x7fa4580601e0 2024-01-21T20:06:50.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.945+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4480095f0 con 0x7fa4580601e0 2024-01-21T20:06:50.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.945+0000 7fa43f7fe700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa44002ff60 0x7fa440032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:50.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.945+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa448040fc0 con 0x7fa4580601e0 2024-01-21T20:06:50.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.945+0000 7fa4567fc700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa44002ff60 0x7fa440032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:50.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.946+0000 7fa4567fc700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa44002ff60 0x7fa440032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa44c0096a0 tx=0x7fa44c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:50.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:50.950+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa448014420 con 0x7fa4580601e0 2024-01-21T20:06:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:50.719012+0000 mgr.smithi119.rqacbm (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:51.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.159+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa458107ed0 con 0x7fa4580601e0 2024-01-21T20:06:51.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.159+0000 7fa43f7fe700 1 -- 172.21.15.153:0/1017246072 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa448010230 con 0x7fa4580601e0 2024-01-21T20:06:51.150 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:51.150 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:51.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa44002ff60 msgr2=0x7fa440032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:51.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa44002ff60 0x7fa440032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa44c0096a0 tx=0x7fa44c006b40).stop 2024-01-21T20:06:51.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 msgr2=0x7fa458106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa458106500 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa448013510 tx=0x7fa44800b040).stop 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 shutdown_connections 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa44002ff60 0x7fa440032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 --2- 172.21.15.153:0/1017246072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4580601e0 0x7fa458106500 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 >> 172.21.15.153:0/1017246072 conn(0x7fa4580f09a0 msgr2=0x7fa4580f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 shutdown_connections 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:51.161+0000 7fa457fff700 1 -- 172.21.15.153:0/1017246072 wait complete. 2024-01-21T20:06:51.153 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:52.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:51.160903+0000 mon.smithi119 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.153:0/1017246072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:52.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:52.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:53.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:52.719382+0000 mgr.smithi119.rqacbm (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:53.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 -- 172.21.15.153:0/1444360350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 msgr2=0x7f19780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:53.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 --2- 172.21.15.153:0/1444360350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19780f66c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1968009fb0 tx=0x7f1968009b30).stop 2024-01-21T20:06:53.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 -- 172.21.15.153:0/1444360350 shutdown_connections 2024-01-21T20:06:53.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 --2- 172.21.15.153:0/1444360350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19780f66c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 -- 172.21.15.153:0/1444360350 >> 172.21.15.153:0/1444360350 conn(0x7f19780f1aa0 msgr2=0x7f19780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 -- 172.21.15.153:0/1444360350 shutdown_connections 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.737+0000 7f197d20d700 1 -- 172.21.15.153:0/1444360350 wait complete. 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f197d20d700 1 Processor -- start 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f197d20d700 1 -- start start 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f197d20d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19781064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f197d20d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19781069a0 con 0x7f19780f62a0 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f1977fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19781064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f1977fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19781064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34966/0 (socket says 172.21.15.153:34966) 2024-01-21T20:06:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.738+0000 7f1977fff700 1 -- 172.21.15.153:0/2349022912 learned_addr learned my addr 172.21.15.153:0/2349022912 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f1977fff700 1 -- 172.21.15.153:0/2349022912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968005040 con 0x7f19780f62a0 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f1977fff700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19781064b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1968013510 tx=0x7f196800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1968009320 con 0x7f19780f62a0 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1968013f00 con 0x7f19780f62a0 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1978106ba0 con 0x7f19780f62a0 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1968014dc0 con 0x7f19780f62a0 2024-01-21T20:06:53.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.739+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1978107030 con 0x7f19780f62a0 2024-01-21T20:06:53.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.740+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1968014420 con 0x7f19780f62a0 2024-01-21T20:06:53.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.740+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1978045ad0 con 0x7f19780f62a0 2024-01-21T20:06:53.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.740+0000 7f1974ff9700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f196002ff60 0x7f1960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:53.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.740+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19680410f0 con 0x7f19780f62a0 2024-01-21T20:06:53.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.740+0000 7f19777fe700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f196002ff60 0x7f1960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:53.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.741+0000 7f19777fe700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f196002ff60 0x7f1960032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f196c0096a0 tx=0x7f196c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:53.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.745+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19680095f0 con 0x7f19780f62a0 2024-01-21T20:06:53.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.950+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1978107ee0 con 0x7f19780f62a0 2024-01-21T20:06:53.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.950+0000 7f1974ff9700 1 -- 172.21.15.153:0/2349022912 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f196801cb10 con 0x7f19780f62a0 2024-01-21T20:06:53.941 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:53.942 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f196002ff60 msgr2=0x7f1960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f196002ff60 0x7f1960032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f196c0096a0 tx=0x7f196c006b40).stop 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 msgr2=0x7f19781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19781064b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1968013510 tx=0x7f196800b040).stop 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 shutdown_connections 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f196002ff60 0x7f1960032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 --2- 172.21.15.153:0/2349022912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19780f62a0 0x7f19781064b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 >> 172.21.15.153:0/2349022912 conn(0x7f19780f1aa0 msgr2=0x7f19780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:53.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 shutdown_connections 2024-01-21T20:06:53.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:53.952+0000 7f197d20d700 1 -- 172.21.15.153:0/2349022912 wait complete. 2024-01-21T20:06:53.945 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:54.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:53.951953+0000 mon.smithi119 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.153:0/2349022912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:55.264 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:55.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:55.337 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:55 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:54.719797+0000 mgr.smithi119.rqacbm (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 -- 172.21.15.153:0/781644414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 msgr2=0x7f515c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 --2- 172.21.15.153:0/781644414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c0f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f514c009fb0 tx=0x7f514c009b30).stop 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 -- 172.21.15.153:0/781644414 shutdown_connections 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 --2- 172.21.15.153:0/781644414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c0f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 -- 172.21.15.153:0/781644414 >> 172.21.15.153:0/781644414 conn(0x7f515c0f0b70 msgr2=0x7f515c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 -- 172.21.15.153:0/781644414 shutdown_connections 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.450+0000 7f516092a700 1 -- 172.21.15.153:0/781644414 wait complete. 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.451+0000 7f516092a700 1 Processor -- start 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.451+0000 7f516092a700 1 -- start start 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.451+0000 7f516092a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:56.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.451+0000 7f516092a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c060280 con 0x7f515c0f5410 2024-01-21T20:06:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f515a59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f515a59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34974/0 (socket says 172.21.15.153:34974) 2024-01-21T20:06:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f515a59c700 1 -- 172.21.15.153:0/3755489361 learned_addr learned my addr 172.21.15.153:0/3755489361 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f515a59c700 1 -- 172.21.15.153:0/3755489361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c005040 con 0x7f515c0f5410 2024-01-21T20:06:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f515a59c700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c109390 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f514c013510 tx=0x7f514c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f514c009320 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f514c013f00 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c060480 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f514c014d90 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.452+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c109ac0 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.453+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f514c0095f0 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.453+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c045ad0 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.453+0000 7f5152ffd700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f514802ff60 0x7f5148032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.453+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f514c041000 con 0x7f515c0f5410 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.454+0000 7f5159d9b700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f514802ff60 0x7f5148032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.454+0000 7f5159d9b700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f514802ff60 0x7f5148032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f51440096a0 tx=0x7f5144006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.458+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f514c014420 con 0x7f515c0f5410 2024-01-21T20:06:56.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.667+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f515c0fae60 con 0x7f515c0f5410 2024-01-21T20:06:56.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.668+0000 7f5152ffd700 1 -- 172.21.15.153:0/3755489361 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f514c01d7f0 con 0x7f515c0f5410 2024-01-21T20:06:56.659 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:56.659 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:56.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f514802ff60 msgr2=0x7f5148032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:56.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f514802ff60 0x7f5148032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f51440096a0 tx=0x7f5144006b40).stop 2024-01-21T20:06:56.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 msgr2=0x7f515c109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:56.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c109390 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f514c013510 tx=0x7f514c00b040).stop 2024-01-21T20:06:56.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 shutdown_connections 2024-01-21T20:06:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f514802ff60 0x7f5148032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 --2- 172.21.15.153:0/3755489361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515c0f5410 0x7f515c109390 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 >> 172.21.15.153:0/3755489361 conn(0x7f515c0f0b70 msgr2=0x7f515c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 shutdown_connections 2024-01-21T20:06:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:56.669+0000 7f516092a700 1 -- 172.21.15.153:0/3755489361 wait complete. 2024-01-21T20:06:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:06:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:56.669320+0000 mon.smithi119 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.153:0/3755489361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:06:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:56.720128+0000 mgr.smithi119.rqacbm (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:58.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:06:58.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:06:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:06:58.720460+0000 mgr.smithi119.rqacbm (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:06:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:58.786727+0000 mon.smithi119 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:06:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:58.786998+0000 mon.smithi119 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 -- 172.21.15.153:0/155997302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 msgr2=0x7fdc3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 --2- 172.21.15.153:0/155997302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c0f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fdc2c009fb0 tx=0x7fdc2c009b30).stop 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 -- 172.21.15.153:0/155997302 shutdown_connections 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 --2- 172.21.15.153:0/155997302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c0f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 -- 172.21.15.153:0/155997302 >> 172.21.15.153:0/155997302 conn(0x7fdc3c0f0b70 msgr2=0x7fdc3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 -- 172.21.15.153:0/155997302 shutdown_connections 2024-01-21T20:06:59.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 -- 172.21.15.153:0/155997302 wait complete. 2024-01-21T20:06:59.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.171+0000 7fdc43dce700 1 Processor -- start 2024-01-21T20:06:59.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc43dce700 1 -- start start 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc43dce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc43dce700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c106c00 con 0x7fdc3c0f5410 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc41b6a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc41b6a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:34996/0 (socket says 172.21.15.153:34996) 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc41b6a700 1 -- 172.21.15.153:0/1801347943 learned_addr learned my addr 172.21.15.153:0/1801347943 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc41b6a700 1 -- 172.21.15.153:0/1801347943 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c005040 con 0x7fdc3c0f5410 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc41b6a700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c106710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fdc2c013510 tx=0x7fdc2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2c009320 con 0x7fdc3c0f5410 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.172+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c106e00 con 0x7fdc3c0f5410 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.173+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc2c013f00 con 0x7fdc3c0f5410 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.173+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc2c014d90 con 0x7fdc3c0f5410 2024-01-21T20:06:59.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.173+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c107290 con 0x7fdc3c0f5410 2024-01-21T20:06:59.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.173+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc2c0095f0 con 0x7fdc3c0f5410 2024-01-21T20:06:59.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.173+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c045ad0 con 0x7fdc3c0f5410 2024-01-21T20:06:59.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.173+0000 7fdc327fc700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc2802ff60 0x7fdc28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:06:59.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.174+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc2c041090 con 0x7fdc3c0f5410 2024-01-21T20:06:59.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.174+0000 7fdc41369700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc2802ff60 0x7fdc28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:06:59.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.174+0000 7fdc41369700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc2802ff60 0x7fdc28032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdc380096a0 tx=0x7fdc38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:06:59.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.178+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc2c014420 con 0x7fdc3c0f5410 2024-01-21T20:06:59.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.384+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc3c108140 con 0x7fdc3c0f5410 2024-01-21T20:06:59.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.385+0000 7fdc327fc700 1 -- 172.21.15.153:0/1801347943 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc2c01d7f0 con 0x7fdc3c0f5410 2024-01-21T20:06:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:06:59.376 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:06:59.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc2802ff60 msgr2=0x7fdc28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc2802ff60 0x7fdc28032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdc380096a0 tx=0x7fdc38006b40).stop 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 msgr2=0x7fdc3c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c106710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fdc2c013510 tx=0x7fdc2c00b040).stop 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 shutdown_connections 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdc2802ff60 0x7fdc28032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 --2- 172.21.15.153:0/1801347943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdc3c0f5410 0x7fdc3c106710 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 >> 172.21.15.153:0/1801347943 conn(0x7fdc3c0f0b70 msgr2=0x7fdc3c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 shutdown_connections 2024-01-21T20:06:59.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:06:59.387+0000 7fdc43dce700 1 -- 172.21.15.153:0/1801347943 wait complete. 2024-01-21T20:06:59.379 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:00.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:06:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:06:59.386620+0000 mon.smithi119 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.153:0/1801347943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:00.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:00.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:01.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:00.720759+0000 mgr.smithi119.rqacbm (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 -- 172.21.15.153:0/1743747409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 msgr2=0x7f98400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 --2- 172.21.15.153:0/1743747409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f98400f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f9830009fb0 tx=0x7f9830009b30).stop 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 -- 172.21.15.153:0/1743747409 shutdown_connections 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 --2- 172.21.15.153:0/1743747409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f98400f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 -- 172.21.15.153:0/1743747409 >> 172.21.15.153:0/1743747409 conn(0x7f98400f0b70 msgr2=0x7f98400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 -- 172.21.15.153:0/1743747409 shutdown_connections 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.957+0000 7f98454e3700 1 -- 172.21.15.153:0/1743747409 wait complete. 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f98454e3700 1 Processor -- start 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f98454e3700 1 -- start start 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f98454e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f9840108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f98454e3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9840108e60 con 0x7f98400f5410 2024-01-21T20:07:01.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f983effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f9840108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f983effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f9840108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:35016/0 (socket says 172.21.15.153:35016) 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.958+0000 7f983effd700 1 -- 172.21.15.153:0/2661282493 learned_addr learned my addr 172.21.15.153:0/2661282493 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f983effd700 1 -- 172.21.15.153:0/2661282493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830005040 con 0x7f98400f5410 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f983effd700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f9840108970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9830009f80 tx=0x7f983000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9830009320 con 0x7f98400f5410 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9830013ec0 con 0x7f98400f5410 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9840109060 con 0x7f98400f5410 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f983001ece0 con 0x7f98400f5410 2024-01-21T20:07:01.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.959+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98401094f0 con 0x7f98400f5410 2024-01-21T20:07:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.960+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f983001e420 con 0x7f98400f5410 2024-01-21T20:07:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.960+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9840045ad0 con 0x7f98400f5410 2024-01-21T20:07:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.960+0000 7f98277fe700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f982802ff60 0x7f9828032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.960+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9830040ee0 con 0x7f98400f5410 2024-01-21T20:07:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.960+0000 7f983e7fc700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f982802ff60 0x7f9828032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:01.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.961+0000 7f983e7fc700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f982802ff60 0x7f9828032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f98340096a0 tx=0x7f9834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:01.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:01.965+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f983001e730 con 0x7f98400f5410 2024-01-21T20:07:02.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.170+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f984010a3a0 con 0x7f98400f5410 2024-01-21T20:07:02.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.171+0000 7f98277fe700 1 -- 172.21.15.153:0/2661282493 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f983001cb10 con 0x7f98400f5410 2024-01-21T20:07:02.162 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:02.162 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:02.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f982802ff60 msgr2=0x7f9828032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:02.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f982802ff60 0x7f9828032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f98340096a0 tx=0x7f9834006b40).stop 2024-01-21T20:07:02.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 msgr2=0x7f9840108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:02.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f9840108970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9830009f80 tx=0x7f983000b040).stop 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 shutdown_connections 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f982802ff60 0x7f9828032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 --2- 172.21.15.153:0/2661282493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98400f5410 0x7f9840108970 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 >> 172.21.15.153:0/2661282493 conn(0x7f98400f0b70 msgr2=0x7f984005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 shutdown_connections 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:02.172+0000 7f98454e3700 1 -- 172.21.15.153:0/2661282493 wait complete. 2024-01-21T20:07:02.164 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:02.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:02.172171+0000 mon.smithi119 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.153:0/2661282493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:03.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:03.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:03.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:02.721122+0000 mgr.smithi119.rqacbm (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- 172.21.15.153:0/1518916208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 msgr2=0x7ff2ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 --2- 172.21.15.153:0/1518916208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac0f90f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7ff2a8009fb0 tx=0x7ff2a8009b30).stop 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- 172.21.15.153:0/1518916208 shutdown_connections 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 --2- 172.21.15.153:0/1518916208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac0f90f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- 172.21.15.153:0/1518916208 >> 172.21.15.153:0/1518916208 conn(0x7ff2ac0f0b70 msgr2=0x7ff2ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- 172.21.15.153:0/1518916208 shutdown_connections 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- 172.21.15.153:0/1518916208 wait complete. 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 Processor -- start 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- start start 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.749+0000 7ff2b33f8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac106bb0 con 0x7ff2ac060280 2024-01-21T20:07:04.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b1194700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b1194700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37724/0 (socket says 172.21.15.153:37724) 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b1194700 1 -- 172.21.15.153:0/4198484071 learned_addr learned my addr 172.21.15.153:0/4198484071 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b1194700 1 -- 172.21.15.153:0/4198484071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a8005040 con 0x7ff2ac060280 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b1194700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac1066c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff2a8013510 tx=0x7ff2a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a8009320 con 0x7ff2ac060280 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac106db0 con 0x7ff2ac060280 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2a8013f00 con 0x7ff2ac060280 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2a8014dc0 con 0x7ff2ac060280 2024-01-21T20:07:04.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.750+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac107240 con 0x7ff2ac060280 2024-01-21T20:07:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.751+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2a801d3d0 con 0x7ff2ac060280 2024-01-21T20:07:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.751+0000 7ff2a1ffb700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff29802ffb0 0x7ff298032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.751+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2a8041120 con 0x7ff2ac060280 2024-01-21T20:07:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.751+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac045ad0 con 0x7ff2ac060280 2024-01-21T20:07:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.752+0000 7ff2b0993700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff29802ffb0 0x7ff298032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.752+0000 7ff2b0993700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff29802ffb0 0x7ff298032470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff29c0096a0 tx=0x7ff29c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:04.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.756+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2a8012920 con 0x7ff2ac060280 2024-01-21T20:07:04.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.962+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2ac0fc040 con 0x7ff2ac060280 2024-01-21T20:07:04.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.963+0000 7ff2a1ffb700 1 -- 172.21.15.153:0/4198484071 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2a802b020 con 0x7ff2ac060280 2024-01-21T20:07:04.953 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:04.954 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:04.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.964+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff29802ffb0 msgr2=0x7ff298032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:04.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.964+0000 7ff2b33f8700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff29802ffb0 0x7ff298032470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7ff29c0096a0 tx=0x7ff29c006b40).stop 2024-01-21T20:07:04.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.964+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 msgr2=0x7ff2ac1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:04.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.964+0000 7ff2b33f8700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac1066c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7ff2a8013510 tx=0x7ff2a800b040).stop 2024-01-21T20:07:04.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.964+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 shutdown_connections 2024-01-21T20:07:04.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.965+0000 7ff2b33f8700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff29802ffb0 0x7ff298032470 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:04.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.965+0000 7ff2b33f8700 1 --2- 172.21.15.153:0/4198484071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ac060280 0x7ff2ac1066c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:04.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.965+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 >> 172.21.15.153:0/4198484071 conn(0x7ff2ac0f0b70 msgr2=0x7ff2ac0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:04.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.965+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 shutdown_connections 2024-01-21T20:07:04.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:04.965+0000 7ff2b33f8700 1 -- 172.21.15.153:0/4198484071 wait complete. 2024-01-21T20:07:04.956 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:05.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:04.721378+0000 mgr.smithi119.rqacbm (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:06.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:05 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:04.964222+0000 mon.smithi119 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.153:0/4198484071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:06.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:06.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:07.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 -- 172.21.15.153:0/4142614650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 msgr2=0x7fde9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 --2- 172.21.15.153:0/4142614650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fde8c009fb0 tx=0x7fde8c009b30).stop 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 -- 172.21.15.153:0/4142614650 shutdown_connections 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 --2- 172.21.15.153:0/4142614650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 -- 172.21.15.153:0/4142614650 >> 172.21.15.153:0/4142614650 conn(0x7fde9c0f0b70 msgr2=0x7fde9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 -- 172.21.15.153:0/4142614650 shutdown_connections 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.500+0000 7fdea3b0e700 1 -- 172.21.15.153:0/4142614650 wait complete. 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea3b0e700 1 Processor -- start 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea3b0e700 1 -- start start 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea3b0e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea3b0e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9c0fb820 con 0x7fde9c0f5410 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea18aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea18aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37738/0 (socket says 172.21.15.153:37738) 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.501+0000 7fdea18aa700 1 -- 172.21.15.153:0/2215035453 learned_addr learned my addr 172.21.15.153:0/2215035453 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fdea18aa700 1 -- 172.21.15.153:0/2215035453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c005040 con 0x7fde9c0f5410 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fdea18aa700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0fd3a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fde8c013510 tx=0x7fde8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8c009320 con 0x7fde9c0f5410 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9c0fb9c0 con 0x7fde9c0f5410 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde8c013f00 con 0x7fde9c0f5410 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8c014d90 con 0x7fde9c0f5410 2024-01-21T20:07:07.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.502+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9c0fbe50 con 0x7fde9c0f5410 2024-01-21T20:07:07.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.503+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde8c0095f0 con 0x7fde9c0f5410 2024-01-21T20:07:07.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.503+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9c059100 con 0x7fde9c0f5410 2024-01-21T20:07:07.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.503+0000 7fde927fc700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde8802ff60 0x7fde88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:07.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.503+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde8c0410b0 con 0x7fde9c0f5410 2024-01-21T20:07:07.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.503+0000 7fdea10a9700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde8802ff60 0x7fde88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:07.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.504+0000 7fdea10a9700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde8802ff60 0x7fde88032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fde980096a0 tx=0x7fde98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:07.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.508+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde8c014420 con 0x7fde9c0f5410 2024-01-21T20:07:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:06.721737+0000 mgr.smithi119.rqacbm (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:07.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.714+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde9c0fae60 con 0x7fde9c0f5410 2024-01-21T20:07:07.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.714+0000 7fde927fc700 1 -- 172.21.15.153:0/2215035453 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fde8c010230 con 0x7fde9c0f5410 2024-01-21T20:07:07.705 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:07.705 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde8802ff60 msgr2=0x7fde88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde8802ff60 0x7fde88032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fde980096a0 tx=0x7fde98006b40).stop 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 msgr2=0x7fde9c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0fd3a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fde8c013510 tx=0x7fde8c00b040).stop 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 shutdown_connections 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fde8802ff60 0x7fde88032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 --2- 172.21.15.153:0/2215035453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fde9c0f5410 0x7fde9c0fd3a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 >> 172.21.15.153:0/2215035453 conn(0x7fde9c0f0b70 msgr2=0x7fde9c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 shutdown_connections 2024-01-21T20:07:07.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:07.716+0000 7fdea3b0e700 1 -- 172.21.15.153:0/2215035453 wait complete. 2024-01-21T20:07:07.708 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:08.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:07.715734+0000 mon.smithi119 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.153:0/2215035453' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:09.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:09.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:09.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:08.722074+0000 mgr.smithi119.rqacbm (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:10.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 -- 172.21.15.153:0/1485983367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 msgr2=0x7f97240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 --2- 172.21.15.153:0/1485983367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97240f66c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f9714009fb0 tx=0x7f9714009b30).stop 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 -- 172.21.15.153:0/1485983367 shutdown_connections 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 --2- 172.21.15.153:0/1485983367 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97240f66c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 -- 172.21.15.153:0/1485983367 >> 172.21.15.153:0/1485983367 conn(0x7f97240f1aa0 msgr2=0x7f97240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 -- 172.21.15.153:0/1485983367 shutdown_connections 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.302+0000 7f9723fff700 1 -- 172.21.15.153:0/1485983367 wait complete. 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9723fff700 1 Processor -- start 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9723fff700 1 -- start start 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9723fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9723fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97241069a0 con 0x7f97240f62a0 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9722ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9722ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37756/0 (socket says 172.21.15.153:37756) 2024-01-21T20:07:10.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9722ffd700 1 -- 172.21.15.153:0/616939125 learned_addr learned my addr 172.21.15.153:0/616939125 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.303+0000 7f9722ffd700 1 -- 172.21.15.153:0/616939125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714005040 con 0x7f97240f62a0 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.304+0000 7f9722ffd700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97241064b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f9714009830 tx=0x7f971400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.304+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9714009320 con 0x7f97240f62a0 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.304+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9724106ba0 con 0x7f97240f62a0 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.304+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97140136c0 con 0x7f97240f62a0 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.304+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f971401ee90 con 0x7f97240f62a0 2024-01-21T20:07:10.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.304+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9724107030 con 0x7f97240f62a0 2024-01-21T20:07:10.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.305+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f971401e420 con 0x7f97240f62a0 2024-01-21T20:07:10.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.305+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724045ad0 con 0x7f97240f62a0 2024-01-21T20:07:10.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.305+0000 7f970b7fe700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f970c02ff10 0x7f970c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:10.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.305+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9714040f40 con 0x7f97240f62a0 2024-01-21T20:07:10.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.305+0000 7f97227fc700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f970c02ff10 0x7f970c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:10.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.305+0000 7f97227fc700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f970c02ff10 0x7f970c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f97180096a0 tx=0x7f9718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:10.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.309+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f971406d2b0 con 0x7f97240f62a0 2024-01-21T20:07:10.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.514+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9724107ee0 con 0x7f97240f62a0 2024-01-21T20:07:10.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.515+0000 7f970b7fe700 1 -- 172.21.15.153:0/616939125 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f97140095f0 con 0x7f97240f62a0 2024-01-21T20:07:10.506 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:10.506 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f970c02ff10 msgr2=0x7f970c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f970c02ff10 0x7f970c0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f97180096a0 tx=0x7f9718006b40).stop 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 msgr2=0x7f97241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97241064b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f9714009830 tx=0x7f971400b040).stop 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 shutdown_connections 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f970c02ff10 0x7f970c0323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 --2- 172.21.15.153:0/616939125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97240f62a0 0x7f97241064b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 >> 172.21.15.153:0/616939125 conn(0x7f97240f1aa0 msgr2=0x7f97240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 shutdown_connections 2024-01-21T20:07:10.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:10.517+0000 7f9723fff700 1 -- 172.21.15.153:0/616939125 wait complete. 2024-01-21T20:07:10.509 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:10.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:10.516596+0000 mon.smithi119 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.153:0/616939125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:11.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:11 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:10.722409+0000 mgr.smithi119.rqacbm (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:11.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:11.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:13.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 -- 172.21.15.153:0/3911810241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 msgr2=0x7fdd3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:13.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 --2- 172.21.15.153:0/3911810241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fdd30009fb0 tx=0x7fdd30009b30).stop 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 -- 172.21.15.153:0/3911810241 shutdown_connections 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 --2- 172.21.15.153:0/3911810241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 -- 172.21.15.153:0/3911810241 >> 172.21.15.153:0/3911810241 conn(0x7fdd3c0f0b70 msgr2=0x7fdd3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 -- 172.21.15.153:0/3911810241 shutdown_connections 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 -- 172.21.15.153:0/3911810241 wait complete. 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.093+0000 7fdd41df0700 1 Processor -- start 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd41df0700 1 -- start start 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd41df0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd41df0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c106c10 con 0x7fdd3c0f5410 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37772/0 (socket says 172.21.15.153:37772) 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd3b7fe700 1 -- 172.21.15.153:0/488334191 learned_addr learned my addr 172.21.15.153:0/488334191 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.094+0000 7fdd3b7fe700 1 -- 172.21.15.153:0/488334191 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd30005040 con 0x7fdd3c0f5410 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.095+0000 7fdd3b7fe700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c106720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fdd30014000 tx=0x7fdd3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:13.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.095+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd300095f0 con 0x7fdd3c0f5410 2024-01-21T20:07:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.095+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c106e10 con 0x7fdd3c0f5410 2024-01-21T20:07:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.095+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd30007e70 con 0x7fdd3c0f5410 2024-01-21T20:07:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.095+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd30014ea0 con 0x7fdd3c0f5410 2024-01-21T20:07:13.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.095+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c1072a0 con 0x7fdd3c0f5410 2024-01-21T20:07:13.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:12.722801+0000 mgr.smithi119.rqacbm (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:13.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.096+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd30025070 con 0x7fdd3c0f5410 2024-01-21T20:07:13.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.096+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c045ad0 con 0x7fdd3c0f5410 2024-01-21T20:07:13.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.096+0000 7fdd23fff700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd2402ff60 0x7fdd24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:13.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.096+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd30040d40 con 0x7fdd3c0f5410 2024-01-21T20:07:13.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.096+0000 7fdd3affd700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd2402ff60 0x7fdd24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:13.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.096+0000 7fdd3affd700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd2402ff60 0x7fdd24032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fdd2c0096a0 tx=0x7fdd2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:13.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.101+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd3000f710 con 0x7fdd3c0f5410 2024-01-21T20:07:13.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.305+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd3c108150 con 0x7fdd3c0f5410 2024-01-21T20:07:13.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.305+0000 7fdd23fff700 1 -- 172.21.15.153:0/488334191 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd30012330 con 0x7fdd3c0f5410 2024-01-21T20:07:13.296 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:13.296 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:13.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd2402ff60 msgr2=0x7fdd24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd2402ff60 0x7fdd24032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fdd2c0096a0 tx=0x7fdd2c006b40).stop 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 msgr2=0x7fdd3c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c106720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fdd30014000 tx=0x7fdd3000b040).stop 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 shutdown_connections 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdd2402ff60 0x7fdd24032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 --2- 172.21.15.153:0/488334191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd3c0f5410 0x7fdd3c106720 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 >> 172.21.15.153:0/488334191 conn(0x7fdd3c0f0b70 msgr2=0x7fdd3c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 shutdown_connections 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:13.307+0000 7fdd41df0700 1 -- 172.21.15.153:0/488334191 wait complete. 2024-01-21T20:07:13.298 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:14.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:13.306723+0000 mon.smithi119 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.153:0/488334191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:14.683 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:14.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:15.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:14.723106+0000 mgr.smithi119.rqacbm (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 -- 172.21.15.153:0/2093476569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 msgr2=0x7f36740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 --2- 172.21.15.153:0/2093476569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36740f55c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f3670009fb0 tx=0x7f3670009b30).stop 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 -- 172.21.15.153:0/2093476569 shutdown_connections 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 --2- 172.21.15.153:0/2093476569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36740f55c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 -- 172.21.15.153:0/2093476569 >> 172.21.15.153:0/2093476569 conn(0x7f36740f09a0 msgr2=0x7f36740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 -- 172.21.15.153:0/2093476569 shutdown_connections 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.916+0000 7f367a157700 1 -- 172.21.15.153:0/2093476569 wait complete. 2024-01-21T20:07:15.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f367a157700 1 Processor -- start 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f367a157700 1 -- start start 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f367a157700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36741086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f367a157700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3674108ba0 con 0x7f36740f51a0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f3679155700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36741086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f3679155700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36741086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37340/0 (socket says 172.21.15.153:37340) 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f3679155700 1 -- 172.21.15.153:0/1918488398 learned_addr learned my addr 172.21.15.153:0/1918488398 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.917+0000 7f3679155700 1 -- 172.21.15.153:0/1918488398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670005040 con 0x7f36740f51a0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.918+0000 7f3679155700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36741086b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f3670009830 tx=0x7f367000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.918+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3670009320 con 0x7f36740f51a0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.918+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3674108da0 con 0x7f36740f51a0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.918+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36700136c0 con 0x7f36740f51a0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.918+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f367001ee90 con 0x7f36740f51a0 2024-01-21T20:07:15.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.918+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3674109230 con 0x7f36740f51a0 2024-01-21T20:07:15.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.919+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f367001e420 con 0x7f36740f51a0 2024-01-21T20:07:15.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.919+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3674045ad0 con 0x7f36740f51a0 2024-01-21T20:07:15.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.919+0000 7f3669ffb700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f366002ff00 0x7f36600323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:15.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.919+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3670040fe0 con 0x7f36740f51a0 2024-01-21T20:07:15.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.919+0000 7f3678954700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f366002ff00 0x7f36600323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:15.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.919+0000 7f3678954700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f366002ff00 0x7f36600323c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f36640096a0 tx=0x7f3664006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:15.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:15.923+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f36700095f0 con 0x7f36740f51a0 2024-01-21T20:07:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.126+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f367410a0e0 con 0x7f36740f51a0 2024-01-21T20:07:16.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.127+0000 7f3669ffb700 1 -- 172.21.15.153:0/1918488398 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f367002c2d0 con 0x7f36740f51a0 2024-01-21T20:07:16.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:16.117 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f366002ff00 msgr2=0x7f36600323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f366002ff00 0x7f36600323c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f36640096a0 tx=0x7f3664006b40).stop 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 msgr2=0x7f36741086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36741086b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f3670009830 tx=0x7f367000b040).stop 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 shutdown_connections 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f366002ff00 0x7f36600323c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 --2- 172.21.15.153:0/1918488398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36740f51a0 0x7f36741086b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 >> 172.21.15.153:0/1918488398 conn(0x7f36740f09a0 msgr2=0x7f36740f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:16.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 shutdown_connections 2024-01-21T20:07:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:16.128+0000 7f367a157700 1 -- 172.21.15.153:0/1918488398 wait complete. 2024-01-21T20:07:16.120 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:16.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:16 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:16.127249+0000 mon.smithi119 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.153:0/1918488398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:17.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:17.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:17.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:16.723458+0000 mgr.smithi119.rqacbm (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:18.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 -- 172.21.15.153:0/2543568426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 msgr2=0x7f72400f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:18.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 --2- 172.21.15.153:0/2543568426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f72400f4fa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f7230009fb0 tx=0x7f7230009b30).stop 2024-01-21T20:07:18.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 -- 172.21.15.153:0/2543568426 shutdown_connections 2024-01-21T20:07:18.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 --2- 172.21.15.153:0/2543568426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f72400f4fa0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:18.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 -- 172.21.15.153:0/2543568426 >> 172.21.15.153:0/2543568426 conn(0x7f72400f02e0 msgr2=0x7f72400f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:18.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 -- 172.21.15.153:0/2543568426 shutdown_connections 2024-01-21T20:07:18.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.717+0000 7f72483f7700 1 -- 172.21.15.153:0/2543568426 wait complete. 2024-01-21T20:07:18.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f72483f7700 1 Processor -- start 2024-01-21T20:07:18.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f72483f7700 1 -- start start 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f72483f7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f7240109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f72483f7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72400f18f0 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f7246193700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f7240109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f7246193700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f7240109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37356/0 (socket says 172.21.15.153:37356) 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.718+0000 7f7246193700 1 -- 172.21.15.153:0/1687981350 learned_addr learned my addr 172.21.15.153:0/1687981350 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f7246193700 1 -- 172.21.15.153:0/1687981350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7230005040 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f7246193700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f7240109410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f7230013510 tx=0x7f723000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7230009320 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7230013f00 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72400f1af0 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7230014d90 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.719+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7240109ac0 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.720+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72300095f0 con 0x7f72400f4b80 2024-01-21T20:07:18.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.720+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7240059100 con 0x7f72400f4b80 2024-01-21T20:07:18.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.720+0000 7f7236ffd700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f722c02ff60 0x7f722c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:18.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.720+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7230041000 con 0x7f72400f4b80 2024-01-21T20:07:18.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.720+0000 7f7245992700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f722c02ff60 0x7f722c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:18.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.721+0000 7f7245992700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f722c02ff60 0x7f722c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f723c0096a0 tx=0x7f723c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:18.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.725+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7230014420 con 0x7f72400f4b80 2024-01-21T20:07:18.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.927+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7240045ad0 con 0x7f72400f4b80 2024-01-21T20:07:18.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.928+0000 7f7236ffd700 1 -- 172.21.15.153:0/1687981350 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f723001d7f0 con 0x7f72400f4b80 2024-01-21T20:07:18.918 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:18.918 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f722c02ff60 msgr2=0x7f722c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f722c02ff60 0x7f722c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f723c0096a0 tx=0x7f723c006b40).stop 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 msgr2=0x7f7240109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f7240109410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f7230013510 tx=0x7f723000b040).stop 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 shutdown_connections 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f722c02ff60 0x7f722c032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 --2- 172.21.15.153:0/1687981350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72400f4b80 0x7f7240109410 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 >> 172.21.15.153:0/1687981350 conn(0x7f72400f02e0 msgr2=0x7f72400f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:18.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 shutdown_connections 2024-01-21T20:07:18.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:18.929+0000 7f72483f7700 1 -- 172.21.15.153:0/1687981350 wait complete. 2024-01-21T20:07:18.920 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:19.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:18.723733+0000 mgr.smithi119.rqacbm (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:20.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:18.928164+0000 mon.smithi119 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.153:0/1687981350' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:20.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:20.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:21.088 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:20.724078+0000 mgr.smithi119.rqacbm (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:21.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 -- 172.21.15.153:0/1295475460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 msgr2=0x7f2e900f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:21.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 --2- 172.21.15.153:0/1295475460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e900f7dc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f2e80009fb0 tx=0x7f2e80009b30).stop 2024-01-21T20:07:21.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 -- 172.21.15.153:0/1295475460 shutdown_connections 2024-01-21T20:07:21.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 --2- 172.21.15.153:0/1295475460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e900f7dc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:21.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 -- 172.21.15.153:0/1295475460 >> 172.21.15.153:0/1295475460 conn(0x7f2e900f0120 msgr2=0x7f2e900f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 -- 172.21.15.153:0/1295475460 shutdown_connections 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 -- 172.21.15.153:0/1295475460 wait complete. 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 Processor -- start 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.501+0000 7f2e965ea700 1 -- start start 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e965ea700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e901064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e965ea700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e901069a0 con 0x7f2e900f79a0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e955e8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e901064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e955e8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e901064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37384/0 (socket says 172.21.15.153:37384) 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e955e8700 1 -- 172.21.15.153:0/2179650908 learned_addr learned my addr 172.21.15.153:0/2179650908 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e955e8700 1 -- 172.21.15.153:0/2179650908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80005040 con 0x7f2e900f79a0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e955e8700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e901064b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2e80013510 tx=0x7f2e8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e80009320 con 0x7f2e900f79a0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e90106ba0 con 0x7f2e900f79a0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e80013f00 con 0x7f2e900f79a0 2024-01-21T20:07:21.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.502+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e80014dc0 con 0x7f2e900f79a0 2024-01-21T20:07:21.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.503+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e90107030 con 0x7f2e900f79a0 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.503+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e800095f0 con 0x7f2e900f79a0 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.504+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90045ad0 con 0x7f2e900f79a0 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.504+0000 7f2e85ffb700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2e7c02ffb0 0x7f2e7c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.504+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e80041100 con 0x7f2e900f79a0 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.504+0000 7f2e94de7700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2e7c02ffb0 0x7f2e7c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.504+0000 7f2e94de7700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2e7c02ffb0 0x7f2e7c032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2e8c0096a0 tx=0x7f2e8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:21.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.508+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e80014420 con 0x7f2e900f79a0 2024-01-21T20:07:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.713+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e90107ee0 con 0x7f2e900f79a0 2024-01-21T20:07:21.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.714+0000 7f2e85ffb700 1 -- 172.21.15.153:0/2179650908 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2e80010230 con 0x7f2e900f79a0 2024-01-21T20:07:21.704 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:21.704 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2e7c02ffb0 msgr2=0x7f2e7c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2e7c02ffb0 0x7f2e7c032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2e8c0096a0 tx=0x7f2e8c006b40).stop 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 msgr2=0x7f2e901064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e901064b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f2e80013510 tx=0x7f2e8000b040).stop 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 shutdown_connections 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f2e7c02ffb0 0x7f2e7c032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 --2- 172.21.15.153:0/2179650908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e900f79a0 0x7f2e901064b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 >> 172.21.15.153:0/2179650908 conn(0x7f2e900f0120 msgr2=0x7f2e900f0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 shutdown_connections 2024-01-21T20:07:21.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:21.715+0000 7f2e965ea700 1 -- 172.21.15.153:0/2179650908 wait complete. 2024-01-21T20:07:21.706 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:22.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:21.714156+0000 mon.smithi119 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.153:0/2179650908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:23.103 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:23.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:23.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:22.724494+0000 mgr.smithi119.rqacbm (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:24.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 -- 172.21.15.153:0/3647883617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540f5410 msgr2=0x7f12540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 --2- 172.21.15.153:0/3647883617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540f5410 0x7f12540f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1244009fb0 tx=0x7f1244009b30).stop 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 -- 172.21.15.153:0/3647883617 shutdown_connections 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 --2- 172.21.15.153:0/3647883617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540f5410 0x7f12540f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 -- 172.21.15.153:0/3647883617 >> 172.21.15.153:0/3647883617 conn(0x7f12540f0b70 msgr2=0x7f12540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 -- 172.21.15.153:0/3647883617 shutdown_connections 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.245+0000 7f125b45a700 1 -- 172.21.15.153:0/3647883617 wait complete. 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f125b45a700 1 Processor -- start 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f125b45a700 1 -- start start 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f125b45a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 0x7f12541095a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f125b45a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1254109a90 con 0x7f12540601e0 2024-01-21T20:07:24.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f12591f6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 0x7f12541095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:24.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f12591f6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 0x7f12541095a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54086/0 (socket says 172.21.15.153:54086) 2024-01-21T20:07:24.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.246+0000 7f12591f6700 1 -- 172.21.15.153:0/1044648197 learned_addr learned my addr 172.21.15.153:0/1044648197 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:24.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f12591f6700 1 -- 172.21.15.153:0/1044648197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1244005040 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f12591f6700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 0x7f12541095a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1244009f80 tx=0x7f1244009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12440136c0 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1254109bd0 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12440095f0 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1244014420 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.247+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125410a000 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.248+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1244025060 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.248+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1254045ad0 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.248+0000 7f1249ffb700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f124002ff60 0x7f1240032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.248+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1244040d80 con 0x7f12540601e0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.248+0000 7f12589f5700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f124002ff60 0x7f1240032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:24.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.249+0000 7f12589f5700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f124002ff60 0x7f1240032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f12500096a0 tx=0x7f1250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:24.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.253+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f124401cb60 con 0x7f12540601e0 2024-01-21T20:07:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.463+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12540fae60 con 0x7f12540601e0 2024-01-21T20:07:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.464+0000 7f1249ffb700 1 -- 172.21.15.153:0/1044648197 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1244042070 con 0x7f12540601e0 2024-01-21T20:07:24.454 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:24.454 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f124002ff60 msgr2=0x7f1240032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f124002ff60 0x7f1240032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f12500096a0 tx=0x7f1250006b40).stop 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 msgr2=0x7f12541095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 0x7f12541095a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1244009f80 tx=0x7f1244009b30).stop 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 shutdown_connections 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f124002ff60 0x7f1240032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 --2- 172.21.15.153:0/1044648197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12540601e0 0x7f12541095a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 >> 172.21.15.153:0/1044648197 conn(0x7f12540f0b70 msgr2=0x7f12540f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 shutdown_connections 2024-01-21T20:07:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:24.466+0000 7f125b45a700 1 -- 172.21.15.153:0/1044648197 wait complete. 2024-01-21T20:07:24.457 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:24.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:24.464663+0000 mon.smithi119 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.153:0/1044648197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:25.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:25.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:25.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:24.724789+0000 mgr.smithi119.rqacbm (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.004+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/396491815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 msgr2=0x7f8eb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.004+0000 7f8ebaa8b700 1 --2- 172.21.15.153:0/396491815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb40f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f8ea8009fb0 tx=0x7f8ea8009b30).stop 2024-01-21T20:07:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.004+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/396491815 shutdown_connections 2024-01-21T20:07:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.004+0000 7f8ebaa8b700 1 --2- 172.21.15.153:0/396491815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb40f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:26.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.004+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/396491815 >> 172.21.15.153:0/396491815 conn(0x7f8eb40f0b70 msgr2=0x7f8eb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.005+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/396491815 shutdown_connections 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.005+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/396491815 wait complete. 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.005+0000 7f8ebaa8b700 1 Processor -- start 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.005+0000 7f8ebaa8b700 1 -- start start 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.005+0000 7f8ebaa8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.005+0000 7f8ebaa8b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb4105b20 con 0x7f8eb40f5410 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb8827700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb8827700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54102/0 (socket says 172.21.15.153:54102) 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb8827700 1 -- 172.21.15.153:0/187124372 learned_addr learned my addr 172.21.15.153:0/187124372 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb8827700 1 -- 172.21.15.153:0/187124372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea8005040 con 0x7f8eb40f5410 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb8827700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb4105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f8ea8013510 tx=0x7f8ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea8009320 con 0x7f8eb40f5410 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ea8013f00 con 0x7f8eb40f5410 2024-01-21T20:07:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4105d20 con 0x7f8eb40f5410 2024-01-21T20:07:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ea8014d90 con 0x7f8eb40f5410 2024-01-21T20:07:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.006+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb41061b0 con 0x7f8eb40f5410 2024-01-21T20:07:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.007+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ea80263a0 con 0x7f8eb40f5410 2024-01-21T20:07:26.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.007+0000 7f8eb17fa700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8ea402ff60 0x7f8ea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.007+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ea8041ff0 con 0x7f8eb40f5410 2024-01-21T20:07:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.007+0000 7f8eb3fff700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8ea402ff60 0x7f8ea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.007+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb4059100 con 0x7f8eb40f5410 2024-01-21T20:07:27.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.008+0000 7f8eb3fff700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8ea402ff60 0x7f8ea4032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8ea00096a0 tx=0x7f8ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:27.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.012+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ea8012920 con 0x7f8eb40f5410 2024-01-21T20:07:27.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.222+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8eb40fae60 con 0x7f8eb40f5410 2024-01-21T20:07:27.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.222+0000 7f8eb17fa700 1 -- 172.21.15.153:0/187124372 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ea802b020 con 0x7f8eb40f5410 2024-01-21T20:07:27.216 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:27.216 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:27.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8ea402ff60 msgr2=0x7f8ea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:27.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8ea402ff60 0x7f8ea4032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f8ea00096a0 tx=0x7f8ea0006b40).stop 2024-01-21T20:07:27.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 msgr2=0x7f8eb4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb4105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f8ea8013510 tx=0x7f8ea800b040).stop 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 shutdown_connections 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f8ea402ff60 0x7f8ea4032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 --2- 172.21.15.153:0/187124372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8eb40f5410 0x7f8eb4105630 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 >> 172.21.15.153:0/187124372 conn(0x7f8eb40f0b70 msgr2=0x7f8eb40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 shutdown_connections 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:27.224+0000 7f8ebaa8b700 1 -- 172.21.15.153:0/187124372 wait complete. 2024-01-21T20:07:27.218 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:27.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:26.725144+0000 mgr.smithi119.rqacbm (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:28.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:28.586 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:28.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:07:26.887220+0000 mgr.smithi119.rqacbm (mgr.14162) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi119 ... 2024-01-21T20:07:28.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Verifying port 9100 ... 2024-01-21T20:07:28.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Non-zero exit code 1 from systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119 2024-01-21T20:07:28.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: systemctl: stderr Job for ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119.service failed because a timeout was exceeded. 2024-01-21T20:07:28.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: systemctl: stderr See "systemctl status ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119.service" and "journalctl -xe" for details. 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Traceback (most recent call last): 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: main() 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: r = ctx.func(ctx) 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: return func(ctx) 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ports=daemon_ports) 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T20:07:28.588 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: RuntimeError: Failed command: systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Traceback (most recent call last): 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: yield (conn, connr) 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: code, '\n'.join(err))) 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi119 ... 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Verifying port 9100 ... 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Non-zero exit code 1 from systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: systemctl: stderr Job for ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119.service failed because a timeout was exceeded. 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: systemctl: stderr See "systemctl status ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119.service" and "journalctl -xe" for details. 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: Traceback (most recent call last): 2024-01-21T20:07:28.589 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: main() 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: r = ctx.func(ctx) 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: return func(ctx) 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: ports=daemon_ports) 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: File "/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: RuntimeError: Failed command: systemctl start ceph-79b26e34-b897-11ee-95b0-87774f69a715@node-exporter.smithi119 2024-01-21T20:07:28.590 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cephadm 2024-01-21T20:07:26.894028+0000 mgr.smithi119.rqacbm (mgr.14162) 232 : cephadm [INF] Deploying daemon prometheus.smithi119 on smithi119 2024-01-21T20:07:28.591 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:27.223813+0000 mon.smithi119 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.153:0/187124372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:29.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:28.725526+0000 mgr.smithi119.rqacbm (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 -- 172.21.15.153:0/2705908556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 msgr2=0x7fd9dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 --2- 172.21.15.153:0/2705908556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc0f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd9d0009fb0 tx=0x7fd9d0009b30).stop 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 -- 172.21.15.153:0/2705908556 shutdown_connections 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 --2- 172.21.15.153:0/2705908556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc0f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 -- 172.21.15.153:0/2705908556 >> 172.21.15.153:0/2705908556 conn(0x7fd9dc0f0b70 msgr2=0x7fd9dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 -- 172.21.15.153:0/2705908556 shutdown_connections 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.766+0000 7fd9e3205700 1 -- 172.21.15.153:0/2705908556 wait complete. 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e3205700 1 Processor -- start 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e3205700 1 -- start start 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e3205700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:29.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e3205700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc060220 con 0x7fd9dc0f5410 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e0fa1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e0fa1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54118/0 (socket says 172.21.15.153:54118) 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.767+0000 7fd9e0fa1700 1 -- 172.21.15.153:0/2045687082 learned_addr learned my addr 172.21.15.153:0/2045687082 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9e0fa1700 1 -- 172.21.15.153:0/2045687082 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0005040 con 0x7fd9dc0f5410 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9e0fa1700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd9d0013510 tx=0x7fd9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d0009320 con 0x7fd9dc0f5410 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc060420 con 0x7fd9dc0f5410 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9d0013f00 con 0x7fd9dc0f5410 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d0014d90 con 0x7fd9dc0f5410 2024-01-21T20:07:29.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.768+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc109b00 con 0x7fd9dc0f5410 2024-01-21T20:07:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.769+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9d00263a0 con 0x7fd9dc0f5410 2024-01-21T20:07:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.769+0000 7fd9d97fa700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.769+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9d00410a0 con 0x7fd9dc0f5410 2024-01-21T20:07:29.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.769+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc059100 con 0x7fd9dc0f5410 2024-01-21T20:07:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.769+0000 7fd9dbfff700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:29.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.769+0000 7fd9dbfff700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9cc02ff60 0x7fd9cc032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd9c80096a0 tx=0x7fd9c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:29.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.774+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9d0012920 con 0x7fd9dc0f5410 2024-01-21T20:07:29.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.979+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9dc045ad0 con 0x7fd9dc0f5410 2024-01-21T20:07:29.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.980+0000 7fd9d97fa700 1 -- 172.21.15.153:0/2045687082 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd9d002b020 con 0x7fd9dc0f5410 2024-01-21T20:07:29.971 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:29.971 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.981+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9cc02ff60 msgr2=0x7fd9cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.981+0000 7fd9e3205700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9cc02ff60 0x7fd9cc032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd9c80096a0 tx=0x7fd9c8006b40).stop 2024-01-21T20:07:29.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.981+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 msgr2=0x7fd9dc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.981+0000 7fd9e3205700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd9d0013510 tx=0x7fd9d000b040).stop 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.982+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 shutdown_connections 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.982+0000 7fd9e3205700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.982+0000 7fd9e3205700 1 --2- 172.21.15.153:0/2045687082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9dc0f5410 0x7fd9dc109450 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.982+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 >> 172.21.15.153:0/2045687082 conn(0x7fd9dc0f0b70 msgr2=0x7fd9dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.982+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 shutdown_connections 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:29.982+0000 7fd9e3205700 1 -- 172.21.15.153:0/2045687082 wait complete. 2024-01-21T20:07:29.973 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:30.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:29.981618+0000 mon.smithi119 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.153:0/2045687082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:31.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:31.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:31.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:31 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:30.725880+0000 mgr.smithi119.rqacbm (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 -- 172.21.15.153:0/993502402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 msgr2=0x7f4ebc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 --2- 172.21.15.153:0/993502402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc0f66c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f4eac009fb0 tx=0x7f4eac009b30).stop 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 -- 172.21.15.153:0/993502402 shutdown_connections 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 --2- 172.21.15.153:0/993502402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc0f66c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 -- 172.21.15.153:0/993502402 >> 172.21.15.153:0/993502402 conn(0x7f4ebc0f1aa0 msgr2=0x7f4ebc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 -- 172.21.15.153:0/993502402 shutdown_connections 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.573+0000 7f4ec0c02700 1 -- 172.21.15.153:0/993502402 wait complete. 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ec0c02700 1 Processor -- start 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ec0c02700 1 -- start start 2024-01-21T20:07:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ec0c02700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ec0c02700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc1069a0 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ebb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ebb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:54128/0 (socket says 172.21.15.153:54128) 2024-01-21T20:07:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ebb7fe700 1 -- 172.21.15.153:0/3788808679 learned_addr learned my addr 172.21.15.153:0/3788808679 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:32.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.574+0000 7f4ebb7fe700 1 -- 172.21.15.153:0/3788808679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac005040 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.575+0000 7f4ebb7fe700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc1064b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4eac013510 tx=0x7f4eac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:32.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.575+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4eac009320 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.575+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc106ba0 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.575+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4eac013f00 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.575+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4eac014dc0 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.575+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc107030 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.576+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4eac014420 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.576+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc045ad0 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.576+0000 7f4eb3fff700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4ea802ff60 0x7f4ea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.576+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4eac041020 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.576+0000 7f4ebaffd700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4ea802ff60 0x7f4ea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.576+0000 7f4ebaffd700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4ea802ff60 0x7f4ea8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4ea40096a0 tx=0x7f4ea4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:32.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.581+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4eac0095f0 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.785+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ebc107e60 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.786+0000 7f4eb3fff700 1 -- 172.21.15.153:0/3788808679 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4eac01cb10 con 0x7f4ebc0f62a0 2024-01-21T20:07:32.777 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:32.777 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4ea802ff60 msgr2=0x7f4ea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4ea802ff60 0x7f4ea8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4ea40096a0 tx=0x7f4ea4006b40).stop 2024-01-21T20:07:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 msgr2=0x7f4ebc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc1064b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4eac013510 tx=0x7f4eac00b040).stop 2024-01-21T20:07:32.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 shutdown_connections 2024-01-21T20:07:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f4ea802ff60 0x7f4ea8032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 --2- 172.21.15.153:0/3788808679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ebc0f62a0 0x7f4ebc1064b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 >> 172.21.15.153:0/3788808679 conn(0x7f4ebc0f1aa0 msgr2=0x7f4ebc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 shutdown_connections 2024-01-21T20:07:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:32.787+0000 7f4ec0c02700 1 -- 172.21.15.153:0/3788808679 wait complete. 2024-01-21T20:07:32.779 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:33.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:32.726265+0000 mgr.smithi119.rqacbm (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:33.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:32.787335+0000 mon.smithi119 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.153:0/3788808679' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:34.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:34.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:35.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:34.726616+0000 mgr.smithi119.rqacbm (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.482+0000 7f4615ceb700 1 -- 172.21.15.153:0/629632419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 msgr2=0x7f46100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.482+0000 7f4615ceb700 1 --2- 172.21.15.153:0/629632419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46100f55c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f4604009fb0 tx=0x7f4604009b30).stop 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.482+0000 7f4615ceb700 1 -- 172.21.15.153:0/629632419 shutdown_connections 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.482+0000 7f4615ceb700 1 --2- 172.21.15.153:0/629632419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46100f55c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.482+0000 7f4615ceb700 1 -- 172.21.15.153:0/629632419 >> 172.21.15.153:0/629632419 conn(0x7f46100f09a0 msgr2=0x7f46100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4615ceb700 1 -- 172.21.15.153:0/629632419 shutdown_connections 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4615ceb700 1 -- 172.21.15.153:0/629632419 wait complete. 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4615ceb700 1 Processor -- start 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4615ceb700 1 -- start start 2024-01-21T20:07:35.475 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4615ceb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46101053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4615ceb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46101058a0 con 0x7f46100f51a0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.483+0000 7f4614ce9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46101053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f4614ce9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46101053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39450/0 (socket says 172.21.15.153:39450) 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f4614ce9700 1 -- 172.21.15.153:0/669419008 learned_addr learned my addr 172.21.15.153:0/669419008 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f4614ce9700 1 -- 172.21.15.153:0/669419008 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4604005040 con 0x7f46100f51a0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f4614ce9700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46101053b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f4604013510 tx=0x7f460400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4604009320 con 0x7f46100f51a0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4610105aa0 con 0x7f46100f51a0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4604013f00 con 0x7f46100f51a0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4604014dc0 con 0x7f46100f51a0 2024-01-21T20:07:35.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.484+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4610105f30 con 0x7f46100f51a0 2024-01-21T20:07:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.485+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4604014420 con 0x7f46100f51a0 2024-01-21T20:07:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.485+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4610045ad0 con 0x7f46100f51a0 2024-01-21T20:07:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.485+0000 7f460d7fa700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f460002ff60 0x7f4600032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:35.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.485+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4604041030 con 0x7f46100f51a0 2024-01-21T20:07:35.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.485+0000 7f460ffff700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f460002ff60 0x7f4600032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:35.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.486+0000 7f460ffff700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f460002ff60 0x7f4600032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f45fc0096a0 tx=0x7f45fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:35.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.490+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46040095f0 con 0x7f46100f51a0 2024-01-21T20:07:35.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.694+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4610106de0 con 0x7f46100f51a0 2024-01-21T20:07:35.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.695+0000 7f460d7fa700 1 -- 172.21.15.153:0/669419008 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f460401cb10 con 0x7f46100f51a0 2024-01-21T20:07:35.686 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:35.686 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.696+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f460002ff60 msgr2=0x7f4600032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.696+0000 7f4615ceb700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f460002ff60 0x7f4600032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f45fc0096a0 tx=0x7f45fc006b40).stop 2024-01-21T20:07:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.696+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 msgr2=0x7f46101053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.696+0000 7f4615ceb700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46101053b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f4604013510 tx=0x7f460400b040).stop 2024-01-21T20:07:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.697+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 shutdown_connections 2024-01-21T20:07:35.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.697+0000 7f4615ceb700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f460002ff60 0x7f4600032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.697+0000 7f4615ceb700 1 --2- 172.21.15.153:0/669419008 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46100f51a0 0x7f46101053b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.697+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 >> 172.21.15.153:0/669419008 conn(0x7f46100f09a0 msgr2=0x7f46100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.697+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 shutdown_connections 2024-01-21T20:07:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:35.697+0000 7f4615ceb700 1 -- 172.21.15.153:0/669419008 wait complete. 2024-01-21T20:07:35.688 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:36.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:35.696347+0000 mon.smithi119 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.153:0/669419008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:37.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:37.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:36.726913+0000 mgr.smithi119.rqacbm (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:38.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 -- 172.21.15.153:0/126890336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 msgr2=0x7faa180f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 --2- 172.21.15.153:0/126890336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180f6e20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7faa0c009fb0 tx=0x7faa0c009b30).stop 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 -- 172.21.15.153:0/126890336 shutdown_connections 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 --2- 172.21.15.153:0/126890336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180f6e20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 -- 172.21.15.153:0/126890336 >> 172.21.15.153:0/126890336 conn(0x7faa180f0260 msgr2=0x7faa180f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 -- 172.21.15.153:0/126890336 shutdown_connections 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.171+0000 7faa1f04a700 1 -- 172.21.15.153:0/126890336 wait complete. 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1f04a700 1 Processor -- start 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1f04a700 1 -- start start 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1f04a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180fb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1f04a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180f9a70 con 0x7faa180f6a00 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1cde6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180fb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1cde6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180fb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39480/0 (socket says 172.21.15.153:39480) 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.172+0000 7faa1cde6700 1 -- 172.21.15.153:0/893928991 learned_addr learned my addr 172.21.15.153:0/893928991 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa1cde6700 1 -- 172.21.15.153:0/893928991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c005040 con 0x7faa180f6a00 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa1cde6700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180fb5f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7faa0c009830 tx=0x7faa0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0c009320 con 0x7faa180f6a00 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa180f9c10 con 0x7faa180f6a00 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa0c0136c0 con 0x7faa180f6a00 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0c01ee90 con 0x7faa180f6a00 2024-01-21T20:07:38.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.173+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa180fa0a0 con 0x7faa180f6a00 2024-01-21T20:07:38.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.174+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa0c01e420 con 0x7faa180f6a00 2024-01-21T20:07:38.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.174+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa18059100 con 0x7faa180f6a00 2024-01-21T20:07:38.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.174+0000 7faa157fa700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faa0802ff10 0x7faa080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:38.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.174+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa0c041010 con 0x7faa180f6a00 2024-01-21T20:07:38.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.174+0000 7faa17fff700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faa0802ff10 0x7faa080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:38.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.175+0000 7faa17fff700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faa0802ff10 0x7faa080323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7faa040096a0 tx=0x7faa04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:38.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.179+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa0c014960 con 0x7faa180f6a00 2024-01-21T20:07:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.386+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa180fc000 con 0x7faa180f6a00 2024-01-21T20:07:38.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.386+0000 7faa157fa700 1 -- 172.21.15.153:0/893928991 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faa0c01cb10 con 0x7faa180f6a00 2024-01-21T20:07:38.377 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:38.377 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:38.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faa0802ff10 msgr2=0x7faa080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faa0802ff10 0x7faa080323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7faa040096a0 tx=0x7faa04006b40).stop 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 msgr2=0x7faa180fb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180fb5f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7faa0c009830 tx=0x7faa0c00b040).stop 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 shutdown_connections 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7faa0802ff10 0x7faa080323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 --2- 172.21.15.153:0/893928991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa180f6a00 0x7faa180fb5f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 >> 172.21.15.153:0/893928991 conn(0x7faa180f0260 msgr2=0x7faa180f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 shutdown_connections 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:38.388+0000 7faa1f04a700 1 -- 172.21.15.153:0/893928991 wait complete. 2024-01-21T20:07:38.379 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:38.836 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:38.387798+0000 mon.smithi119 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.153:0/893928991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:39.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:39.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:39.836 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:38.727266+0000 mgr.smithi119.rqacbm (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 -- 172.21.15.153:0/4143250067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 msgr2=0x7f92800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 --2- 172.21.15.153:0/4143250067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f92800f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f9268009fb0 tx=0x7f9268009b30).stop 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 -- 172.21.15.153:0/4143250067 shutdown_connections 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 --2- 172.21.15.153:0/4143250067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f92800f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 -- 172.21.15.153:0/4143250067 >> 172.21.15.153:0/4143250067 conn(0x7f92800f1c70 msgr2=0x7f92800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 -- 172.21.15.153:0/4143250067 shutdown_connections 2024-01-21T20:07:40.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.954+0000 7f9285395700 1 -- 172.21.15.153:0/4143250067 wait complete. 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f9285395700 1 Processor -- start 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f9285395700 1 -- start start 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f9285395700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f928006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f9285395700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9280069e20 con 0x7f92800f6510 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f927effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f928006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f927effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f928006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39496/0 (socket says 172.21.15.153:39496) 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f927effd700 1 -- 172.21.15.153:0/3694622891 learned_addr learned my addr 172.21.15.153:0/3694622891 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.955+0000 7f927effd700 1 -- 172.21.15.153:0/3694622891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9268005040 con 0x7f92800f6510 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.956+0000 7f927effd700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f928006b9a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f9268013510 tx=0x7f926800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:40.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.956+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9268009320 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.956+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280069fc0 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.956+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9268013f00 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.956+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9268014d90 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.956+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928006a450 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.957+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9268014420 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.957+0000 7f92777fe700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f926c02ff60 0x7f926c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.957+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9280059100 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.957+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92680410c0 con 0x7f92800f6510 2024-01-21T20:07:40.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.957+0000 7f927e7fc700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f926c02ff60 0x7f926c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:40.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.958+0000 7f927e7fc700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f926c02ff60 0x7f926c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f92700096a0 tx=0x7f9270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:40.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:40.962+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f926806d2b0 con 0x7f92800f6510 2024-01-21T20:07:41.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:40.727643+0000 mgr.smithi119.rqacbm (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:41.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.167+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92800fbf60 con 0x7f92800f6510 2024-01-21T20:07:41.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.168+0000 7f92777fe700 1 -- 172.21.15.153:0/3694622891 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f926801dd50 con 0x7f92800f6510 2024-01-21T20:07:41.159 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:41.159 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:41.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f926c02ff60 msgr2=0x7f926c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:41.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f926c02ff60 0x7f926c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f92700096a0 tx=0x7f9270006b40).stop 2024-01-21T20:07:41.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 msgr2=0x7f928006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:41.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f928006b9a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f9268013510 tx=0x7f926800b040).stop 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 shutdown_connections 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f926c02ff60 0x7f926c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 --2- 172.21.15.153:0/3694622891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92800f6510 0x7f928006b9a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 >> 172.21.15.153:0/3694622891 conn(0x7f92800f1c70 msgr2=0x7f92800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 shutdown_connections 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:41.170+0000 7f9285395700 1 -- 172.21.15.153:0/3694622891 wait complete. 2024-01-21T20:07:41.161 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:42.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:41 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:41.169460+0000 mon.smithi119 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.153:0/3694622891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:42.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:42.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:43.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:42.728021+0000 mgr.smithi119.rqacbm (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:43.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 -- 172.21.15.153:0/14211793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 msgr2=0x7f0e100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 --2- 172.21.15.153:0/14211793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e100f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0e0c009fb0 tx=0x7f0e0c009b30).stop 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 -- 172.21.15.153:0/14211793 shutdown_connections 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 --2- 172.21.15.153:0/14211793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e100f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 -- 172.21.15.153:0/14211793 >> 172.21.15.153:0/14211793 conn(0x7f0e100f1c70 msgr2=0x7f0e100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 -- 172.21.15.153:0/14211793 shutdown_connections 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.693+0000 7f0e17a84700 1 -- 172.21.15.153:0/14211793 wait complete. 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e17a84700 1 Processor -- start 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e17a84700 1 -- start start 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e17a84700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:43.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e17a84700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e10106c20 con 0x7f0e100f6510 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e15820700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e15820700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:53242/0 (socket says 172.21.15.153:53242) 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e15820700 1 -- 172.21.15.153:0/4069700966 learned_addr learned my addr 172.21.15.153:0/4069700966 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.694+0000 7f0e15820700 1 -- 172.21.15.153:0/4069700966 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c005040 con 0x7f0e100f6510 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.695+0000 7f0e15820700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e10106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f0e0c014000 tx=0x7f0e0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.695+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e0c0095f0 con 0x7f0e100f6510 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.695+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e10106e20 con 0x7f0e100f6510 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.695+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e0c007e70 con 0x7f0e100f6510 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.695+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e0c014ea0 con 0x7f0e100f6510 2024-01-21T20:07:43.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.695+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e101072b0 con 0x7f0e100f6510 2024-01-21T20:07:43.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.696+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e0c025070 con 0x7f0e100f6510 2024-01-21T20:07:43.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.696+0000 7f0e067fc700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0dfc02ff60 0x7f0dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:43.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.696+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e0c040a10 con 0x7f0e100f6510 2024-01-21T20:07:43.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.696+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e10045ad0 con 0x7f0e100f6510 2024-01-21T20:07:43.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.696+0000 7f0e1501f700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0dfc02ff60 0x7f0dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:43.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.696+0000 7f0e1501f700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0dfc02ff60 0x7f0dfc032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0e000096a0 tx=0x7f0e00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:43.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.700+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e0c026350 con 0x7f0e100f6510 2024-01-21T20:07:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.909+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e10107fd0 con 0x7f0e100f6510 2024-01-21T20:07:43.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.910+0000 7f0e067fc700 1 -- 172.21.15.153:0/4069700966 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0e0c012330 con 0x7f0e100f6510 2024-01-21T20:07:43.901 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:43.901 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:43.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0dfc02ff60 msgr2=0x7f0dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:43.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0dfc02ff60 0x7f0dfc032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0e000096a0 tx=0x7f0e00006b40).stop 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 msgr2=0x7f0e10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e10106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f0e0c014000 tx=0x7f0e0c00b040).stop 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 shutdown_connections 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0dfc02ff60 0x7f0dfc032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 --2- 172.21.15.153:0/4069700966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e100f6510 0x7f0e10106730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 >> 172.21.15.153:0/4069700966 conn(0x7f0e100f1c70 msgr2=0x7f0e100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 shutdown_connections 2024-01-21T20:07:43.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:43.912+0000 7f0e17a84700 1 -- 172.21.15.153:0/4069700966 wait complete. 2024-01-21T20:07:43.904 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:44.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:43.911661+0000 mon.smithi119 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.153:0/4069700966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:45.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:45.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:45.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:44.728387+0000 mgr.smithi119.rqacbm (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 -- 172.21.15.153:0/4020503915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 msgr2=0x7f1b380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 --2- 172.21.15.153:0/4020503915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b380f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f1b2c009fb0 tx=0x7f1b2c009b30).stop 2024-01-21T20:07:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 -- 172.21.15.153:0/4020503915 shutdown_connections 2024-01-21T20:07:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 --2- 172.21.15.153:0/4020503915 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b380f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:46.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 -- 172.21.15.153:0/4020503915 >> 172.21.15.153:0/4020503915 conn(0x7f1b380f1c70 msgr2=0x7f1b380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:46.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 -- 172.21.15.153:0/4020503915 shutdown_connections 2024-01-21T20:07:46.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.486+0000 7f1b3eb83700 1 -- 172.21.15.153:0/4020503915 wait complete. 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.487+0000 7f1b3eb83700 1 Processor -- start 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.487+0000 7f1b3eb83700 1 -- start start 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3eb83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3eb83700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38106c20 con 0x7f1b380f6510 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3c91f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3c91f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:53272/0 (socket says 172.21.15.153:53272) 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3c91f700 1 -- 172.21.15.153:0/1698080244 learned_addr learned my addr 172.21.15.153:0/1698080244 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3c91f700 1 -- 172.21.15.153:0/1698080244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c005040 con 0x7f1b380f6510 2024-01-21T20:07:46.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3c91f700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f1b2c013510 tx=0x7f1b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c009320 con 0x7f1b380f6510 2024-01-21T20:07:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.488+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38106e20 con 0x7f1b380f6510 2024-01-21T20:07:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.489+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b2c013f00 con 0x7f1b380f6510 2024-01-21T20:07:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.489+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b2c014d90 con 0x7f1b380f6510 2024-01-21T20:07:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.489+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b381072b0 con 0x7f1b380f6510 2024-01-21T20:07:46.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.489+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b2c0263a0 con 0x7f1b380f6510 2024-01-21T20:07:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.489+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b38045ad0 con 0x7f1b380f6510 2024-01-21T20:07:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.489+0000 7f1b357fa700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1b2802ff60 0x7f1b28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.490+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b2c041310 con 0x7f1b380f6510 2024-01-21T20:07:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.490+0000 7f1b37fff700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1b2802ff60 0x7f1b28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:46.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.490+0000 7f1b37fff700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1b2802ff60 0x7f1b28032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1b240096a0 tx=0x7f1b24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:46.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.494+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b2c01cb50 con 0x7f1b380f6510 2024-01-21T20:07:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.699+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b380fbf60 con 0x7f1b380f6510 2024-01-21T20:07:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.699+0000 7f1b357fa700 1 -- 172.21.15.153:0/1698080244 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1b2c01c970 con 0x7f1b380f6510 2024-01-21T20:07:46.690 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:46.690 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1b2802ff60 msgr2=0x7f1b28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1b2802ff60 0x7f1b28032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1b240096a0 tx=0x7f1b24006b40).stop 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 msgr2=0x7f1b38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f1b2c013510 tx=0x7f1b2c00b040).stop 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 shutdown_connections 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f1b2802ff60 0x7f1b28032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 --2- 172.21.15.153:0/1698080244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1b380f6510 0x7f1b38106730 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 >> 172.21.15.153:0/1698080244 conn(0x7f1b380f1c70 msgr2=0x7f1b380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 shutdown_connections 2024-01-21T20:07:46.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:46.701+0000 7f1b3eb83700 1 -- 172.21.15.153:0/1698080244 wait complete. 2024-01-21T20:07:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:46.700792+0000 mon.smithi119 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.153:0/1698080244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:46.728715+0000 mgr.smithi119.rqacbm (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:48.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:48.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:49.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:48.729062+0000 mgr.smithi119.rqacbm (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.297+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3417677322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 msgr2=0x7fabd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 --2- 172.21.15.153:0/3417677322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd80f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fabc0009fb0 tx=0x7fabc0009b30).stop 2024-01-21T20:07:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3417677322 shutdown_connections 2024-01-21T20:07:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 --2- 172.21.15.153:0/3417677322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd80f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3417677322 >> 172.21.15.153:0/3417677322 conn(0x7fabd80f0b70 msgr2=0x7fabd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:49.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3417677322 shutdown_connections 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3417677322 wait complete. 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 Processor -- start 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 -- start start 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.298+0000 7fabdcb9f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd8106c10 con 0x7fabd80f5410 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabd659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabd659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:53294/0 (socket says 172.21.15.153:53294) 2024-01-21T20:07:49.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabd659c700 1 -- 172.21.15.153:0/3301235733 learned_addr learned my addr 172.21.15.153:0/3301235733 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabd659c700 1 -- 172.21.15.153:0/3301235733 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0005040 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabd659c700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd8106720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fabc0013510 tx=0x7fabc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc0009320 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabc0013f00 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd8106e10 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabc0014d90 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.299+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd81072a0 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.300+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabc00095f0 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.300+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd8045ad0 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.300+0000 7fabceffd700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fabc402ff60 0x7fabc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.300+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabc0040fe0 con 0x7fabd80f5410 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.300+0000 7fabd5d9b700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fabc402ff60 0x7fabc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:49.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.301+0000 7fabd5d9b700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fabc402ff60 0x7fabc4032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fabc80096a0 tx=0x7fabc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:49.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.305+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabc001d7f0 con 0x7fabd80f5410 2024-01-21T20:07:49.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.514+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fabd8108150 con 0x7fabd80f5410 2024-01-21T20:07:49.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.514+0000 7fabceffd700 1 -- 172.21.15.153:0/3301235733 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fabc0010230 con 0x7fabd80f5410 2024-01-21T20:07:49.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:49.505 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fabc402ff60 msgr2=0x7fabc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fabc402ff60 0x7fabc4032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fabc80096a0 tx=0x7fabc8006b40).stop 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 msgr2=0x7fabd8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd8106720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fabc0013510 tx=0x7fabc000b040).stop 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 shutdown_connections 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fabc402ff60 0x7fabc4032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 --2- 172.21.15.153:0/3301235733 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabd80f5410 0x7fabd8106720 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 >> 172.21.15.153:0/3301235733 conn(0x7fabd80f0b70 msgr2=0x7fabd805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 shutdown_connections 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:49.516+0000 7fabdcb9f700 1 -- 172.21.15.153:0/3301235733 wait complete. 2024-01-21T20:07:49.507 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:50.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:49.515723+0000 mon.smithi119 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.153:0/3301235733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:50.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:50.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:51.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:50.729382+0000 mgr.smithi119.rqacbm (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:52.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 -- 172.21.15.153:0/277880229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 msgr2=0x7f434c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:52.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 --2- 172.21.15.153:0/277880229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c0f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f433c009fb0 tx=0x7f433c009b30).stop 2024-01-21T20:07:52.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 -- 172.21.15.153:0/277880229 shutdown_connections 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 --2- 172.21.15.153:0/277880229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c0f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 -- 172.21.15.153:0/277880229 >> 172.21.15.153:0/277880229 conn(0x7f434c0f09a0 msgr2=0x7f434c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 -- 172.21.15.153:0/277880229 shutdown_connections 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 -- 172.21.15.153:0/277880229 wait complete. 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.094+0000 7f43514c6700 1 Processor -- start 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f43514c6700 1 -- start start 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f43514c6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c1086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f43514c6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c108ba0 con 0x7f434c0f51a0 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f434bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c1086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f434bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c1086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:53310/0 (socket says 172.21.15.153:53310) 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f434bfff700 1 -- 172.21.15.153:0/878581512 learned_addr learned my addr 172.21.15.153:0/878581512 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f434bfff700 1 -- 172.21.15.153:0/878581512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c005040 con 0x7f434c0f51a0 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.095+0000 7f434bfff700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c1086b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f433c013510 tx=0x7f433c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:52.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f433c009320 con 0x7f434c0f51a0 2024-01-21T20:07:52.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f433c013f00 con 0x7f434c0f51a0 2024-01-21T20:07:52.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c108da0 con 0x7f434c0f51a0 2024-01-21T20:07:52.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f433c014dc0 con 0x7f434c0f51a0 2024-01-21T20:07:52.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c109230 con 0x7f434c0f51a0 2024-01-21T20:07:52.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c045ad0 con 0x7f434c0f51a0 2024-01-21T20:07:52.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.096+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f433c0095f0 con 0x7f434c0f51a0 2024-01-21T20:07:52.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.097+0000 7f4348ff9700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f433402ff50 0x7f4334032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:52.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.097+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f433c0410c0 con 0x7f434c0f51a0 2024-01-21T20:07:52.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.097+0000 7f434b7fe700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f433402ff50 0x7f4334032410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:52.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.097+0000 7f434b7fe700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f433402ff50 0x7f4334032410 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f43400096a0 tx=0x7f4340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:52.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.101+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f433c014420 con 0x7f434c0f51a0 2024-01-21T20:07:52.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.306+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f434c10a0e0 con 0x7f434c0f51a0 2024-01-21T20:07:52.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.306+0000 7f4348ff9700 1 -- 172.21.15.153:0/878581512 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f433c010230 con 0x7f434c0f51a0 2024-01-21T20:07:52.297 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:52.297 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:52.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f433402ff50 msgr2=0x7f4334032410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:52.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f433402ff50 0x7f4334032410 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f43400096a0 tx=0x7f4340006b40).stop 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 msgr2=0x7f434c1086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c1086b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f433c013510 tx=0x7f433c00b040).stop 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 shutdown_connections 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f433402ff50 0x7f4334032410 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 --2- 172.21.15.153:0/878581512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f434c0f51a0 0x7f434c1086b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 >> 172.21.15.153:0/878581512 conn(0x7f434c0f09a0 msgr2=0x7f434c0f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 shutdown_connections 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:52.308+0000 7f43514c6700 1 -- 172.21.15.153:0/878581512 wait complete. 2024-01-21T20:07:52.299 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:52.836 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:52.307768+0000 mon.smithi119 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.153:0/878581512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:53.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:53.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:53.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:52.729742+0000 mgr.smithi119.rqacbm (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 -- 172.21.15.153:0/2475545772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 msgr2=0x7fdff40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 --2- 172.21.15.153:0/2475545772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff40f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fdfe8009fb0 tx=0x7fdfe8009b30).stop 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 -- 172.21.15.153:0/2475545772 shutdown_connections 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 --2- 172.21.15.153:0/2475545772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff40f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 -- 172.21.15.153:0/2475545772 >> 172.21.15.153:0/2475545772 conn(0x7fdff40f1c70 msgr2=0x7fdff40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 -- 172.21.15.153:0/2475545772 shutdown_connections 2024-01-21T20:07:54.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 -- 172.21.15.153:0/2475545772 wait complete. 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.878+0000 7fdffa06d700 1 Processor -- start 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.879+0000 7fdffa06d700 1 -- start start 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.879+0000 7fdffa06d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.879+0000 7fdffa06d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff4106c20 con 0x7fdff40f6510 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.879+0000 7fdff37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.879+0000 7fdff37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:57072/0 (socket says 172.21.15.153:57072) 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.879+0000 7fdff37fe700 1 -- 172.21.15.153:0/2993154760 learned_addr learned my addr 172.21.15.153:0/2993154760 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:54.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdff37fe700 1 -- 172.21.15.153:0/2993154760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe8005040 con 0x7fdff40f6510 2024-01-21T20:07:54.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdff37fe700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff4106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdfe8013510 tx=0x7fdfe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfe8009320 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfe8013f00 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff4106e20 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfe8014d90 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.880+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff41072b0 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.881+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfe8014420 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.881+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdff4045ad0 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.881+0000 7fdfdbfff700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdfdc02ff60 0x7fdfdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.881+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfe8041000 con 0x7fdff40f6510 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.881+0000 7fdff2ffd700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdfdc02ff60 0x7fdfdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:54.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.882+0000 7fdff2ffd700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdfdc02ff60 0x7fdfdc032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fdfe40096a0 tx=0x7fdfe4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:54.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:54.886+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfe80095f0 con 0x7fdff40f6510 2024-01-21T20:07:55.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.090+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdff40fbf60 con 0x7fdff40f6510 2024-01-21T20:07:55.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.091+0000 7fdfdbfff700 1 -- 172.21.15.153:0/2993154760 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdfe801db70 con 0x7fdff40f6510 2024-01-21T20:07:55.082 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:55.082 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdfdc02ff60 msgr2=0x7fdfdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:55.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdfdc02ff60 0x7fdfdc032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fdfe40096a0 tx=0x7fdfe4006b40).stop 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 msgr2=0x7fdff4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff4106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fdfe8013510 tx=0x7fdfe800b040).stop 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 shutdown_connections 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fdfdc02ff60 0x7fdfdc032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 --2- 172.21.15.153:0/2993154760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdff40f6510 0x7fdff4106730 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 >> 172.21.15.153:0/2993154760 conn(0x7fdff40f1c70 msgr2=0x7fdff40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 shutdown_connections 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:55.093+0000 7fdffa06d700 1 -- 172.21.15.153:0/2993154760 wait complete. 2024-01-21T20:07:55.084 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:55.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:54.730041+0000 mgr.smithi119.rqacbm (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:56.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:55 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:55.092725+0000 mon.smithi119 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.153:0/2993154760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:56.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:56.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:56.730405+0000 mgr.smithi119.rqacbm (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:57.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 -- 172.21.15.153:0/135334870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 msgr2=0x7f35e80f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 --2- 172.21.15.153:0/135334870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e80f6b90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f35d8009fb0 tx=0x7f35d8009b30).stop 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 -- 172.21.15.153:0/135334870 shutdown_connections 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 --2- 172.21.15.153:0/135334870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e80f6b90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 -- 172.21.15.153:0/135334870 >> 172.21.15.153:0/135334870 conn(0x7f35e80f0120 msgr2=0x7f35e80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 -- 172.21.15.153:0/135334870 shutdown_connections 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 -- 172.21.15.153:0/135334870 wait complete. 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.719+0000 7f35ed4d5700 1 Processor -- start 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35ed4d5700 1 -- start start 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35ed4d5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e81064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35ed4d5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e81069c0 con 0x7f35e80f6770 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35e7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e81064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35e7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e81064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:57092/0 (socket says 172.21.15.153:57092) 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35e7fff700 1 -- 172.21.15.153:0/1110122356 learned_addr learned my addr 172.21.15.153:0/1110122356 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:07:57.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35e7fff700 1 -- 172.21.15.153:0/1110122356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8005040 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35e7fff700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e81064d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f35d8014000 tx=0x7f35d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.720+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d80095f0 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e8106bc0 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35d8007e70 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8014ea0 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e8107050 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35d8025070 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e8045ad0 con 0x7f35e80f6770 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.721+0000 7f35e4ff9700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35d002ff60 0x7f35d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:07:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.722+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35d8040920 con 0x7f35e80f6770 2024-01-21T20:07:57.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.722+0000 7f35e77fe700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35d002ff60 0x7f35d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:07:57.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.722+0000 7f35e77fe700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35d002ff60 0x7f35d0032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f35dc0096a0 tx=0x7f35dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:07:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.726+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35d800f4e0 con 0x7f35e80f6770 2024-01-21T20:07:57.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.929+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35e8107e30 con 0x7f35e80f6770 2024-01-21T20:07:57.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.930+0000 7f35e4ff9700 1 -- 172.21.15.153:0/1110122356 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f35d800f300 con 0x7f35e80f6770 2024-01-21T20:07:57.921 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:07:57.921 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:07:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35d002ff60 msgr2=0x7f35d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35d002ff60 0x7f35d0032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f35dc0096a0 tx=0x7f35dc006b40).stop 2024-01-21T20:07:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 msgr2=0x7f35e81064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:07:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e81064d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f35d8014000 tx=0x7f35d800b040).stop 2024-01-21T20:07:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 shutdown_connections 2024-01-21T20:07:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f35d002ff60 0x7f35d0032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 --2- 172.21.15.153:0/1110122356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35e80f6770 0x7f35e81064d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:07:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.931+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 >> 172.21.15.153:0/1110122356 conn(0x7f35e80f0120 msgr2=0x7f35e80f4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:07:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.932+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 shutdown_connections 2024-01-21T20:07:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:07:57.932+0000 7f35ed4d5700 1 -- 172.21.15.153:0/1110122356 wait complete. 2024-01-21T20:07:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:07:58.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:57.931439+0000 mon.smithi119 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.153:0/1110122356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:07:59.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:07:59.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:07:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:07:58.730750+0000 mgr.smithi119.rqacbm (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:07:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:58.787767+0000 mon.smithi119 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:07:59.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:07:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:07:58.787929+0000 mon.smithi119 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- 172.21.15.153:0/2652163599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 msgr2=0x7fb7cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 --2- 172.21.15.153:0/2652163599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc0f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fb7bc009fb0 tx=0x7fb7bc009b30).stop 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- 172.21.15.153:0/2652163599 shutdown_connections 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 --2- 172.21.15.153:0/2652163599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc0f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- 172.21.15.153:0/2652163599 >> 172.21.15.153:0/2652163599 conn(0x7fb7cc0f09a0 msgr2=0x7fb7cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- 172.21.15.153:0/2652163599 shutdown_connections 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- 172.21.15.153:0/2652163599 wait complete. 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 Processor -- start 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- start start 2024-01-21T20:08:00.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.523+0000 7fb7d239c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc060220 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d139a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d139a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:57098/0 (socket says 172.21.15.153:57098) 2024-01-21T20:08:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d139a700 1 -- 172.21.15.153:0/3011317698 learned_addr learned my addr 172.21.15.153:0/3011317698 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d139a700 1 -- 172.21.15.153:0/3011317698 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc005040 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d139a700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc1091d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb7bc013510 tx=0x7fb7bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:00.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7bc009320 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc060420 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7bc013f00 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7bc014dc0 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.524+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc109880 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.525+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7cc059100 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.525+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7bc0095f0 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.525+0000 7fb7c1ffb700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb7b802ff60 0x7fb7b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.525+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7bc041040 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.526+0000 7fb7d0b99700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb7b802ff60 0x7fb7b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:00.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.526+0000 7fb7d0b99700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb7b802ff60 0x7fb7b8032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb7c80096a0 tx=0x7fb7c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:00.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.530+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7bc014420 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.736+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7cc045ad0 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.737+0000 7fb7c1ffb700 1 -- 172.21.15.153:0/3011317698 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7bc010230 con 0x7fb7cc0f51a0 2024-01-21T20:08:00.728 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:00.728 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb7b802ff60 msgr2=0x7fb7b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb7b802ff60 0x7fb7b8032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb7c80096a0 tx=0x7fb7c8006b40).stop 2024-01-21T20:08:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 msgr2=0x7fb7cc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc1091d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb7bc013510 tx=0x7fb7bc00b040).stop 2024-01-21T20:08:00.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 shutdown_connections 2024-01-21T20:08:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fb7b802ff60 0x7fb7b8032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 --2- 172.21.15.153:0/3011317698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7cc0f51a0 0x7fb7cc1091d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 >> 172.21.15.153:0/3011317698 conn(0x7fb7cc0f09a0 msgr2=0x7fb7cc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 shutdown_connections 2024-01-21T20:08:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:00.739+0000 7fb7d239c700 1 -- 172.21.15.153:0/3011317698 wait complete. 2024-01-21T20:08:00.730 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:01.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:00.731065+0000 mgr.smithi119.rqacbm (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:01.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:00.738392+0000 mon.smithi119 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.153:0/3011317698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:02.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:02.069 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:03.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:02.731503+0000 mgr.smithi119.rqacbm (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/4003408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 msgr2=0x7ff0f80fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 --2- 172.21.15.153:0/4003408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f80fb570 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7ff0e8009fb0 tx=0x7ff0e8009b30).stop 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/4003408247 shutdown_connections 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 --2- 172.21.15.153:0/4003408247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f80fb570 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/4003408247 >> 172.21.15.153:0/4003408247 conn(0x7ff0f80f0b70 msgr2=0x7ff0f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/4003408247 shutdown_connections 2024-01-21T20:08:03.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.300+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/4003408247 wait complete. 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0ff8eb700 1 Processor -- start 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0ff8eb700 1 -- start start 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0ff8eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f81088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0ff8eb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8108dc0 con 0x7ff0f8060200 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0fd687700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f81088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0fd687700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f81088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:57116/0 (socket says 172.21.15.153:57116) 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.301+0000 7ff0fd687700 1 -- 172.21.15.153:0/2283743446 learned_addr learned my addr 172.21.15.153:0/2283743446 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0fd687700 1 -- 172.21.15.153:0/2283743446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8005040 con 0x7ff0f8060200 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0fd687700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f81088d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7ff0e8013510 tx=0x7ff0e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e8009320 con 0x7ff0f8060200 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f8108fc0 con 0x7ff0f8060200 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff0e8013f00 con 0x7ff0f8060200 2024-01-21T20:08:03.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e8014dc0 con 0x7ff0f8060200 2024-01-21T20:08:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.302+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f8109450 con 0x7ff0f8060200 2024-01-21T20:08:03.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.303+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0e8014420 con 0x7ff0f8060200 2024-01-21T20:08:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.303+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f8045ad0 con 0x7ff0f8060200 2024-01-21T20:08:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.303+0000 7ff0edffb700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff0e402ff60 0x7ff0e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.303+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0e80410e0 con 0x7ff0f8060200 2024-01-21T20:08:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.303+0000 7ff0fce86700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff0e402ff60 0x7ff0e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:03.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.304+0000 7ff0fce86700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff0e402ff60 0x7ff0e4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff0f40096a0 tx=0x7ff0f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:03.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.308+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0e80095f0 con 0x7ff0f8060200 2024-01-21T20:08:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.511+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0f810a2a0 con 0x7ff0f8060200 2024-01-21T20:08:03.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.511+0000 7ff0edffb700 1 -- 172.21.15.153:0/2283743446 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff0e801cb10 con 0x7ff0f8060200 2024-01-21T20:08:03.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:03.502 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:03.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff0e402ff60 msgr2=0x7ff0e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff0e402ff60 0x7ff0e4032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff0f40096a0 tx=0x7ff0f4006b40).stop 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 msgr2=0x7ff0f81088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f81088d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7ff0e8013510 tx=0x7ff0e800b040).stop 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 shutdown_connections 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7ff0e402ff60 0x7ff0e4032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 --2- 172.21.15.153:0/2283743446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0f8060200 0x7ff0f81088d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 >> 172.21.15.153:0/2283743446 conn(0x7ff0f80f0b70 msgr2=0x7ff0f80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 shutdown_connections 2024-01-21T20:08:03.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:03.513+0000 7ff0ff8eb700 1 -- 172.21.15.153:0/2283743446 wait complete. 2024-01-21T20:08:03.505 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:04.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:03 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:03.512780+0000 mon.smithi119 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.153:0/2283743446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:04.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:04.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:05.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:04 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:04.731879+0000 mgr.smithi119.rqacbm (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:06.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 -- 172.21.15.153:0/932936918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 msgr2=0x7f64c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 --2- 172.21.15.153:0/932936918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c00f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f64b0009fb0 tx=0x7f64b0009b30).stop 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 -- 172.21.15.153:0/932936918 shutdown_connections 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 --2- 172.21.15.153:0/932936918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c00f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 -- 172.21.15.153:0/932936918 >> 172.21.15.153:0/932936918 conn(0x7f64c00f0b70 msgr2=0x7f64c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 -- 172.21.15.153:0/932936918 shutdown_connections 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.151+0000 7f64c82e7700 1 -- 172.21.15.153:0/932936918 wait complete. 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c82e7700 1 Processor -- start 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c82e7700 1 -- start start 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c82e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c0109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:06.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c82e7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0060220 con 0x7f64c00f5410 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c6083700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c0109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c6083700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c0109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:33888/0 (socket says 172.21.15.153:33888) 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.152+0000 7f64c6083700 1 -- 172.21.15.153:0/4154429066 learned_addr learned my addr 172.21.15.153:0/4154429066 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64c6083700 1 -- 172.21.15.153:0/4154429066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0005040 con 0x7f64c00f5410 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64c6083700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c0109420 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f64b0013510 tx=0x7f64b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b0009320 con 0x7f64c00f5410 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c0060420 con 0x7f64c00f5410 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64b0013f00 con 0x7f64c00f5410 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64b0014d90 con 0x7f64c00f5410 2024-01-21T20:08:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.153+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c0109ad0 con 0x7f64c00f5410 2024-01-21T20:08:06.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.154+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64b00095f0 con 0x7f64c00f5410 2024-01-21T20:08:06.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.154+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c0059100 con 0x7f64c00f5410 2024-01-21T20:08:06.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.154+0000 7f64b6ffd700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f64ac02ffb0 0x7f64ac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:06.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.154+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64b0040ff0 con 0x7f64c00f5410 2024-01-21T20:08:06.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.154+0000 7f64c5882700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f64ac02ffb0 0x7f64ac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:06.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.155+0000 7f64c5882700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f64ac02ffb0 0x7f64ac032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f64bc0096a0 tx=0x7f64bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:06.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.159+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64b0014420 con 0x7f64c00f5410 2024-01-21T20:08:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.365+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64c0045ad0 con 0x7f64c00f5410 2024-01-21T20:08:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64b6ffd700 1 -- 172.21.15.153:0/4154429066 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f64b0010230 con 0x7f64c00f5410 2024-01-21T20:08:06.356 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:06.357 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f64ac02ffb0 msgr2=0x7f64ac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f64ac02ffb0 0x7f64ac032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f64bc0096a0 tx=0x7f64bc006b40).stop 2024-01-21T20:08:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 msgr2=0x7f64c0109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c0109420 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f64b0013510 tx=0x7f64b000b040).stop 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 shutdown_connections 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f64ac02ffb0 0x7f64ac032470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 --2- 172.21.15.153:0/4154429066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64c00f5410 0x7f64c0109420 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 >> 172.21.15.153:0/4154429066 conn(0x7f64c00f0b70 msgr2=0x7f64c00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 shutdown_connections 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:06.367+0000 7f64c82e7700 1 -- 172.21.15.153:0/4154429066 wait complete. 2024-01-21T20:08:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:07.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:06.367264+0000 mon.smithi119 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.153:0/4154429066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:07.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:07 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:06.732234+0000 mgr.smithi119.rqacbm (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:07.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:07.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 -- 172.21.15.153:0/1177707695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 msgr2=0x7fa3400f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 --2- 172.21.15.153:0/1177707695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa3400f4f20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fa334009fb0 tx=0x7fa334009b30).stop 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 -- 172.21.15.153:0/1177707695 shutdown_connections 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 --2- 172.21.15.153:0/1177707695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa3400f4f20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 -- 172.21.15.153:0/1177707695 >> 172.21.15.153:0/1177707695 conn(0x7fa3400f0260 msgr2=0x7fa3400f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 -- 172.21.15.153:0/1177707695 shutdown_connections 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 -- 172.21.15.153:0/1177707695 wait complete. 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 Processor -- start 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.922+0000 7fa346e51700 1 -- start start 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa346e51700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa340105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa346e51700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa340105af0 con 0x7fa3400f4b00 2024-01-21T20:08:08.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa344bed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa340105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa344bed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa340105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:33900/0 (socket says 172.21.15.153:33900) 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa344bed700 1 -- 172.21.15.153:0/1753487414 learned_addr learned my addr 172.21.15.153:0/1753487414 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa344bed700 1 -- 172.21.15.153:0/1753487414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa334005040 con 0x7fa3400f4b00 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.923+0000 7fa344bed700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa340105600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa334013510 tx=0x7fa33400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.924+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa334009320 con 0x7fa3400f4b00 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.924+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa334013f00 con 0x7fa3400f4b00 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.924+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa340105cf0 con 0x7fa3400f4b00 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.924+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa334014d90 con 0x7fa3400f4b00 2024-01-21T20:08:08.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.924+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa340106180 con 0x7fa3400f4b00 2024-01-21T20:08:08.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.924+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa33400f410 con 0x7fa3400f4b00 2024-01-21T20:08:08.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.925+0000 7fa33d7fa700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa33002ffb0 0x7fa330032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:08.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.925+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa334042fa0 con 0x7fa3400f4b00 2024-01-21T20:08:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.925+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa340045ad0 con 0x7fa3400f4b00 2024-01-21T20:08:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.925+0000 7fa33ffff700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa33002ffb0 0x7fa330032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.925+0000 7fa33ffff700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa33002ffb0 0x7fa330032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa32c0096a0 tx=0x7fa32c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:08.930+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3340283a0 con 0x7fa3400f4b00 2024-01-21T20:08:09.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:08 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:08.732503+0000 mgr.smithi119.rqacbm (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:09.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.137+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3400ff4f0 con 0x7fa3400f4b00 2024-01-21T20:08:09.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.138+0000 7fa33d7fa700 1 -- 172.21.15.153:0/1753487414 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa33402d020 con 0x7fa3400f4b00 2024-01-21T20:08:09.129 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:09.129 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:09.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa33002ffb0 msgr2=0x7fa330032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa33002ffb0 0x7fa330032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa32c0096a0 tx=0x7fa32c006b40).stop 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 msgr2=0x7fa340105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa340105600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fa334013510 tx=0x7fa33400b040).stop 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 shutdown_connections 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa33002ffb0 0x7fa330032470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 --2- 172.21.15.153:0/1753487414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3400f4b00 0x7fa340105600 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 >> 172.21.15.153:0/1753487414 conn(0x7fa3400f0260 msgr2=0x7fa3400f2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 shutdown_connections 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:09.140+0000 7fa346e51700 1 -- 172.21.15.153:0/1753487414 wait complete. 2024-01-21T20:08:09.131 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:10.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:09 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:09.139638+0000 mon.smithi119 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.153:0/1753487414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:10.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:10.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:11.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:10 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:10.732826+0000 mgr.smithi119.rqacbm (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3237280263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 msgr2=0x7f30a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 --2- 172.21.15.153:0/3237280263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a00f6930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f3094009fb0 tx=0x7f3094009b30).stop 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3237280263 shutdown_connections 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 --2- 172.21.15.153:0/3237280263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a00f6930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3237280263 >> 172.21.15.153:0/3237280263 conn(0x7f30a00f1c70 msgr2=0x7f30a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3237280263 shutdown_connections 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.696+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3237280263 wait complete. 2024-01-21T20:08:11.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f30a5ccf700 1 Processor -- start 2024-01-21T20:08:11.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f30a5ccf700 1 -- start start 2024-01-21T20:08:11.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f30a5ccf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:11.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f30a5ccf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a0069e20 con 0x7f30a00f6510 2024-01-21T20:08:11.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f309f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:11.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f309f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:33916/0 (socket says 172.21.15.153:33916) 2024-01-21T20:08:11.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.697+0000 7f309f7fe700 1 -- 172.21.15.153:0/3408080182 learned_addr learned my addr 172.21.15.153:0/3408080182 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:11.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f309f7fe700 1 -- 172.21.15.153:0/3408080182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094005040 con 0x7f30a00f6510 2024-01-21T20:08:11.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f309f7fe700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a006b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3094013510 tx=0x7f309400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:11.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3094009320 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3094013f00 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a0069fc0 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3094014d90 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.698+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a006a450 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.699+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30940095f0 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.699+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a0059100 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.699+0000 7f3087fff700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f308802ff60 0x7f3088032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.699+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30940410c0 con 0x7f30a00f6510 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.699+0000 7f309effd700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f308802ff60 0x7f3088032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:11.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.700+0000 7f309effd700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f308802ff60 0x7f3088032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f30900096a0 tx=0x7f3090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:11.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.704+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3094014420 con 0x7f30a00f6510 2024-01-21T20:08:11.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.912+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30a00fbf60 con 0x7f30a00f6510 2024-01-21T20:08:11.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.913+0000 7f3087fff700 1 -- 172.21.15.153:0/3408080182 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f309401d7f0 con 0x7f30a00f6510 2024-01-21T20:08:11.903 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:11.903 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:11.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f308802ff60 msgr2=0x7f3088032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:11.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f308802ff60 0x7f3088032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f30900096a0 tx=0x7f3090006b40).stop 2024-01-21T20:08:11.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 msgr2=0x7f30a006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a006b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3094013510 tx=0x7f309400b040).stop 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 shutdown_connections 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f308802ff60 0x7f3088032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 --2- 172.21.15.153:0/3408080182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30a00f6510 0x7f30a006b9a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 >> 172.21.15.153:0/3408080182 conn(0x7f30a00f1c70 msgr2=0x7f30a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 shutdown_connections 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:11.915+0000 7f30a5ccf700 1 -- 172.21.15.153:0/3408080182 wait complete. 2024-01-21T20:08:11.905 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:12.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:12 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:11.913549+0000 mon.smithi119 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.153:0/3408080182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:13.285 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:13.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:13.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:13 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:12.733203+0000 mgr.smithi119.rqacbm (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:14.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1574090729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 msgr2=0x7f0ae40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:14.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 --2- 172.21.15.153:0/1574090729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae40f66c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f0ad8009fb0 tx=0x7f0ad8009b30).stop 2024-01-21T20:08:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1574090729 shutdown_connections 2024-01-21T20:08:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 --2- 172.21.15.153:0/1574090729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae40f66c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1574090729 >> 172.21.15.153:0/1574090729 conn(0x7f0ae40f1aa0 msgr2=0x7f0ae40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1574090729 shutdown_connections 2024-01-21T20:08:14.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1574090729 wait complete. 2024-01-21T20:08:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 Processor -- start 2024-01-21T20:08:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.513+0000 7f0ae9a79700 1 -- start start 2024-01-21T20:08:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae9a79700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae9a79700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae41069a0 con 0x7f0ae40f62a0 2024-01-21T20:08:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae8a77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:14.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae8a77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40382/0 (socket says 172.21.15.153:40382) 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae8a77700 1 -- 172.21.15.153:0/1968443224 learned_addr learned my addr 172.21.15.153:0/1968443224 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae8a77700 1 -- 172.21.15.153:0/1968443224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad8005040 con 0x7f0ae40f62a0 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae8a77700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae41064b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0ad8013510 tx=0x7f0ad800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ad8009320 con 0x7f0ae40f62a0 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae4106ba0 con 0x7f0ae40f62a0 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ad8013f00 con 0x7f0ae40f62a0 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.514+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ad8014dc0 con 0x7f0ae40f62a0 2024-01-21T20:08:14.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.515+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae4107030 con 0x7f0ae40f62a0 2024-01-21T20:08:14.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.515+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ad8014420 con 0x7f0ae40f62a0 2024-01-21T20:08:14.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.515+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae4045ad0 con 0x7f0ae40f62a0 2024-01-21T20:08:14.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.515+0000 7f0ae17fa700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ad402ff60 0x7f0ad4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:14.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.515+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ad8041050 con 0x7f0ae40f62a0 2024-01-21T20:08:14.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.516+0000 7f0ae3fff700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ad402ff60 0x7f0ad4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:14.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.516+0000 7f0ae3fff700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ad402ff60 0x7f0ad4032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0ad00096a0 tx=0x7f0ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:14.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.520+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ad80095f0 con 0x7f0ae40f62a0 2024-01-21T20:08:14.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.725+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ae4107ee0 con 0x7f0ae40f62a0 2024-01-21T20:08:14.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.726+0000 7f0ae17fa700 1 -- 172.21.15.153:0/1968443224 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0ad801cb10 con 0x7f0ae40f62a0 2024-01-21T20:08:14.716 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:14.716 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:14.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ad402ff60 msgr2=0x7f0ad4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:14.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ad402ff60 0x7f0ad4032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0ad00096a0 tx=0x7f0ad0006b40).stop 2024-01-21T20:08:14.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 msgr2=0x7f0ae41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:14.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae41064b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0ad8013510 tx=0x7f0ad800b040).stop 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 shutdown_connections 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f0ad402ff60 0x7f0ad4032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 --2- 172.21.15.153:0/1968443224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ae40f62a0 0x7f0ae41064b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 >> 172.21.15.153:0/1968443224 conn(0x7f0ae40f1aa0 msgr2=0x7f0ae40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 shutdown_connections 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:14.728+0000 7f0ae9a79700 1 -- 172.21.15.153:0/1968443224 wait complete. 2024-01-21T20:08:14.718 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:15.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:14 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:14.726443+0000 mon.smithi119 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.153:0/1968443224' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:16.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:15 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:14.733610+0000 mgr.smithi119.rqacbm (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:16.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:16.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- 172.21.15.153:0/1804460936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 msgr2=0x7f9be00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 --2- 172.21.15.153:0/1804460936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be00f55c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9bd0009fb0 tx=0x7f9bd0009b30).stop 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- 172.21.15.153:0/1804460936 shutdown_connections 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 --2- 172.21.15.153:0/1804460936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be00f55c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- 172.21.15.153:0/1804460936 >> 172.21.15.153:0/1804460936 conn(0x7f9be00f09a0 msgr2=0x7f9be00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- 172.21.15.153:0/1804460936 shutdown_connections 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- 172.21.15.153:0/1804460936 wait complete. 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 Processor -- start 2024-01-21T20:08:17.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- start start 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.300+0000 7f9be6431700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be01058a0 con 0x7f9be00f51a0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be542f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be542f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40394/0 (socket says 172.21.15.153:40394) 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be542f700 1 -- 172.21.15.153:0/2387490813 learned_addr learned my addr 172.21.15.153:0/2387490813 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be542f700 1 -- 172.21.15.153:0/2387490813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0005040 con 0x7f9be00f51a0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be542f700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be01053b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9bd0013510 tx=0x7f9bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd0009320 con 0x7f9be00f51a0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bd0013f00 con 0x7f9be00f51a0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be0105aa0 con 0x7f9be00f51a0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bd0014dc0 con 0x7f9be00f51a0 2024-01-21T20:08:17.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.301+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be0105f30 con 0x7f9be00f51a0 2024-01-21T20:08:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.302+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bd00095f0 con 0x7f9be00f51a0 2024-01-21T20:08:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.302+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be0045ad0 con 0x7f9be00f51a0 2024-01-21T20:08:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.302+0000 7f9bd5ffb700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bcc02ff60 0x7f9bcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:17.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.302+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bd0041020 con 0x7f9be00f51a0 2024-01-21T20:08:17.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.303+0000 7f9be4c2e700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bcc02ff60 0x7f9bcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:17.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.303+0000 7f9be4c2e700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bcc02ff60 0x7f9bcc032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9bdc0096a0 tx=0x7f9bdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:17.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.307+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bd0014420 con 0x7f9be00f51a0 2024-01-21T20:08:17.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.511+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9be0106d80 con 0x7f9be00f51a0 2024-01-21T20:08:17.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.512+0000 7f9bd5ffb700 1 -- 172.21.15.153:0/2387490813 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9bd001c920 con 0x7f9be00f51a0 2024-01-21T20:08:17.502 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:17.502 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bcc02ff60 msgr2=0x7f9bcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bcc02ff60 0x7f9bcc032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9bdc0096a0 tx=0x7f9bdc006b40).stop 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 msgr2=0x7f9be01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be01053b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9bd0013510 tx=0x7f9bd000b040).stop 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 shutdown_connections 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9bcc02ff60 0x7f9bcc032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 --2- 172.21.15.153:0/2387490813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be00f51a0 0x7f9be01053b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 >> 172.21.15.153:0/2387490813 conn(0x7f9be00f09a0 msgr2=0x7f9be00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 shutdown_connections 2024-01-21T20:08:17.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:17.514+0000 7f9be6431700 1 -- 172.21.15.153:0/2387490813 wait complete. 2024-01-21T20:08:17.505 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:17.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:17 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:16.733948+0000 mgr.smithi119.rqacbm (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:18.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:18 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:17.512600+0000 mon.smithi119 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.153:0/2387490813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:18.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:18.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:19.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:19 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:18.734285+0000 mgr.smithi119.rqacbm (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:20.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 -- 172.21.15.153:0/3146291440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 msgr2=0x7f44c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:20.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 --2- 172.21.15.153:0/3146291440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c80f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f44bc009fb0 tx=0x7f44bc009b30).stop 2024-01-21T20:08:20.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 -- 172.21.15.153:0/3146291440 shutdown_connections 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 --2- 172.21.15.153:0/3146291440 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c80f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 -- 172.21.15.153:0/3146291440 >> 172.21.15.153:0/3146291440 conn(0x7f44c80f1c70 msgr2=0x7f44c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 -- 172.21.15.153:0/3146291440 shutdown_connections 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 -- 172.21.15.153:0/3146291440 wait complete. 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.087+0000 7f44ceeab700 1 Processor -- start 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ceeab700 1 -- start start 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ceeab700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ceeab700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8106c20 con 0x7f44c80f6510 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ccc47700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ccc47700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40406/0 (socket says 172.21.15.153:40406) 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ccc47700 1 -- 172.21.15.153:0/828396865 learned_addr learned my addr 172.21.15.153:0/828396865 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.088+0000 7f44ccc47700 1 -- 172.21.15.153:0/828396865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc005040 con 0x7f44c80f6510 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.089+0000 7f44ccc47700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c8106730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f44bc013510 tx=0x7f44bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:20.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.089+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44bc009320 con 0x7f44c80f6510 2024-01-21T20:08:20.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.089+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8106e20 con 0x7f44c80f6510 2024-01-21T20:08:20.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.089+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c81072b0 con 0x7f44c80f6510 2024-01-21T20:08:20.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.089+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44bc013f00 con 0x7f44c80f6510 2024-01-21T20:08:20.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.089+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44bc014d90 con 0x7f44c80f6510 2024-01-21T20:08:20.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.090+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44bc014420 con 0x7f44c80f6510 2024-01-21T20:08:20.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.090+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c8045ad0 con 0x7f44c80f6510 2024-01-21T20:08:20.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.090+0000 7f44c57fa700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44b802ff60 0x7f44b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:20.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.090+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44bc041010 con 0x7f44c80f6510 2024-01-21T20:08:20.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.090+0000 7f44c7fff700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44b802ff60 0x7f44b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:20.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.090+0000 7f44c7fff700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44b802ff60 0x7f44b8032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f44b40096a0 tx=0x7f44b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:20.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.094+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44bc0095f0 con 0x7f44c80f6510 2024-01-21T20:08:20.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.298+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44c81080a0 con 0x7f44c80f6510 2024-01-21T20:08:20.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.299+0000 7f44c57fa700 1 -- 172.21.15.153:0/828396865 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f44bc01db70 con 0x7f44c80f6510 2024-01-21T20:08:20.288 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:20.289 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44b802ff60 msgr2=0x7f44b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44b802ff60 0x7f44b8032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f44b40096a0 tx=0x7f44b4006b40).stop 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 msgr2=0x7f44c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c8106730 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f44bc013510 tx=0x7f44bc00b040).stop 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 shutdown_connections 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f44b802ff60 0x7f44b8032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:20.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 --2- 172.21.15.153:0/828396865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44c80f6510 0x7f44c8106730 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:20.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 >> 172.21.15.153:0/828396865 conn(0x7f44c80f1c70 msgr2=0x7f44c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:20.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.300+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 shutdown_connections 2024-01-21T20:08:20.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:20.301+0000 7f44ceeab700 1 -- 172.21.15.153:0/828396865 wait complete. 2024-01-21T20:08:20.291 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:20.836 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:20 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:20.299164+0000 mon.smithi119 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.153:0/828396865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:21.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:21.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:21.837 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:21 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:20.734674+0000 mgr.smithi119.rqacbm (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:22.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 -- 172.21.15.153:0/2536159576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 msgr2=0x7fc4ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:22.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 --2- 172.21.15.153:0/2536159576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac0f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fc49c009fb0 tx=0x7fc49c009b30).stop 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 -- 172.21.15.153:0/2536159576 shutdown_connections 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 --2- 172.21.15.153:0/2536159576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac0f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 -- 172.21.15.153:0/2536159576 >> 172.21.15.153:0/2536159576 conn(0x7fc4ac0f1c70 msgr2=0x7fc4ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 -- 172.21.15.153:0/2536159576 shutdown_connections 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.889+0000 7fc4b276e700 1 -- 172.21.15.153:0/2536159576 wait complete. 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4b276e700 1 Processor -- start 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4b276e700 1 -- start start 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4b276e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4b276e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac106c20 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4abfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4abfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:40424/0 (socket says 172.21.15.153:40424) 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.890+0000 7fc4abfff700 1 -- 172.21.15.153:0/584677342 learned_addr learned my addr 172.21.15.153:0/584677342 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4abfff700 1 -- 172.21.15.153:0/584677342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c005040 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4abfff700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc49c013510 tx=0x7fc49c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc49c009320 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac106e20 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc49c013f00 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc49c014d90 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.891+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac1072b0 con 0x7fc4ac0f6510 2024-01-21T20:08:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.892+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc49c014420 con 0x7fc4ac0f6510 2024-01-21T20:08:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.892+0000 7fc4a8ff9700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc49402ff60 0x7fc494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.892+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac045ad0 con 0x7fc4ac0f6510 2024-01-21T20:08:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.892+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc49c0410c0 con 0x7fc4ac0f6510 2024-01-21T20:08:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.892+0000 7fc4ab7fe700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc49402ff60 0x7fc494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.893+0000 7fc4ab7fe700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc49402ff60 0x7fc494032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fc4a00096a0 tx=0x7fc4a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:22.897+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc49c0095f0 con 0x7fc4ac0f6510 2024-01-21T20:08:23.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:22 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:22.735070+0000 mgr.smithi119.rqacbm (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.099+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4ac0fc050 con 0x7fc4ac0f6510 2024-01-21T20:08:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.100+0000 7fc4a8ff9700 1 -- 172.21.15.153:0/584677342 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc49c01db70 con 0x7fc4ac0f6510 2024-01-21T20:08:23.090 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:23.090 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc49402ff60 msgr2=0x7fc494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc49402ff60 0x7fc494032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fc4a00096a0 tx=0x7fc4a0006b40).stop 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 msgr2=0x7fc4ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc49c013510 tx=0x7fc49c00b040).stop 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 shutdown_connections 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fc49402ff60 0x7fc494032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 --2- 172.21.15.153:0/584677342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4ac0f6510 0x7fc4ac106730 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 >> 172.21.15.153:0/584677342 conn(0x7fc4ac0f1c70 msgr2=0x7fc4ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 shutdown_connections 2024-01-21T20:08:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:23.101+0000 7fc4b276e700 1 -- 172.21.15.153:0/584677342 wait complete. 2024-01-21T20:08:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:24.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:23 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:23.101263+0000 mon.smithi119 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.153:0/584677342' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:24.467 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:24.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:25.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:24 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:24.735351+0000 mgr.smithi119.rqacbm (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:25.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 -- 172.21.15.153:0/1586586514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 msgr2=0x7f991c0f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 --2- 172.21.15.153:0/1586586514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c0f7dc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f990c009fb0 tx=0x7f990c009b30).stop 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 -- 172.21.15.153:0/1586586514 shutdown_connections 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 --2- 172.21.15.153:0/1586586514 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c0f7dc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 -- 172.21.15.153:0/1586586514 >> 172.21.15.153:0/1586586514 conn(0x7f991c0f0120 msgr2=0x7f991c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 -- 172.21.15.153:0/1586586514 shutdown_connections 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 -- 172.21.15.153:0/1586586514 wait complete. 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.672+0000 7f9921708700 1 Processor -- start 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.673+0000 7f9921708700 1 -- start start 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.673+0000 7f9921708700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.673+0000 7f9921708700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c1069b0 con 0x7f991c0f79a0 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.673+0000 7f991bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:25.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.673+0000 7f991bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37930/0 (socket says 172.21.15.153:37930) 2024-01-21T20:08:25.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.673+0000 7f991bfff700 1 -- 172.21.15.153:0/3939448439 learned_addr learned my addr 172.21.15.153:0/3939448439 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f991bfff700 1 -- 172.21.15.153:0/3939448439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c005040 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f991bfff700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c1064c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f990c013510 tx=0x7f990c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f990c009320 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991c106bb0 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f990c013f00 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f990c014dc0 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.674+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991c107040 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.675+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991c045ad0 con 0x7f991c0f79a0 2024-01-21T20:08:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.675+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f990c01d3d0 con 0x7f991c0f79a0 2024-01-21T20:08:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.675+0000 7f9918ff9700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f990402ff60 0x7f9904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.675+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f990c041340 con 0x7f991c0f79a0 2024-01-21T20:08:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.675+0000 7f991b7fe700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f990402ff60 0x7f9904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:25.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.676+0000 7f991b7fe700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f990402ff60 0x7f9904032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f99100096a0 tx=0x7f9910006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:25.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.680+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f990c0126f0 con 0x7f991c0f79a0 2024-01-21T20:08:25.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.887+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f991c107ef0 con 0x7f991c0f79a0 2024-01-21T20:08:25.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.887+0000 7f9918ff9700 1 -- 172.21.15.153:0/3939448439 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f990c012330 con 0x7f991c0f79a0 2024-01-21T20:08:25.878 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:25.878 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f990402ff60 msgr2=0x7f9904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f990402ff60 0x7f9904032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f99100096a0 tx=0x7f9910006b40).stop 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 msgr2=0x7f991c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c1064c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f990c013510 tx=0x7f990c00b040).stop 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 shutdown_connections 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f990402ff60 0x7f9904032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 --2- 172.21.15.153:0/3939448439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f991c0f79a0 0x7f991c1064c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 >> 172.21.15.153:0/3939448439 conn(0x7f991c0f0120 msgr2=0x7f991c0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 shutdown_connections 2024-01-21T20:08:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:25.889+0000 7f9921708700 1 -- 172.21.15.153:0/3939448439 wait complete. 2024-01-21T20:08:25.881 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:26.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:25 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:25.888943+0000 mon.smithi119 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.153:0/3939448439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:27.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:27.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:27.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:27 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:26.735717+0000 mgr.smithi119.rqacbm (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 -- 172.21.15.153:0/3789103674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 msgr2=0x7f315c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 --2- 172.21.15.153:0/3789103674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c0f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3150009fb0 tx=0x7f3150009b30).stop 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 -- 172.21.15.153:0/3789103674 shutdown_connections 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 --2- 172.21.15.153:0/3789103674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c0f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 -- 172.21.15.153:0/3789103674 >> 172.21.15.153:0/3789103674 conn(0x7f315c0f1c70 msgr2=0x7f315c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 -- 172.21.15.153:0/3789103674 shutdown_connections 2024-01-21T20:08:28.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.445+0000 7f3162bcb700 1 -- 172.21.15.153:0/3789103674 wait complete. 2024-01-21T20:08:28.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3162bcb700 1 Processor -- start 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3162bcb700 1 -- start start 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3162bcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3162bcb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c106c20 con 0x7f315c0f6510 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3160967700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3160967700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37952/0 (socket says 172.21.15.153:37952) 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.446+0000 7f3160967700 1 -- 172.21.15.153:0/335733688 learned_addr learned my addr 172.21.15.153:0/335733688 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f3160967700 1 -- 172.21.15.153:0/335733688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3150005040 con 0x7f315c0f6510 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f3160967700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3150013510 tx=0x7f315000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3150009320 con 0x7f315c0f6510 2024-01-21T20:08:28.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c106e20 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3150013f00 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3150014d90 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.447+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c1072b0 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.448+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3150014420 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.448+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c045ad0 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.448+0000 7f31597fa700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f314c02ff60 0x7f314c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.448+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3150041070 con 0x7f315c0f6510 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.448+0000 7f315bfff700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f314c02ff60 0x7f314c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:28.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.448+0000 7f315bfff700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f314c02ff60 0x7f314c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f31480096a0 tx=0x7f3148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:28.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.452+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31500095f0 con 0x7f315c0f6510 2024-01-21T20:08:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.658+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f315c0fbf60 con 0x7f315c0f6510 2024-01-21T20:08:28.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.659+0000 7f31597fa700 1 -- 172.21.15.153:0/335733688 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f315001db70 con 0x7f315c0f6510 2024-01-21T20:08:28.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:28.650 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.660+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f314c02ff60 msgr2=0x7f314c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.660+0000 7f3162bcb700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f314c02ff60 0x7f314c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f31480096a0 tx=0x7f3148006b40).stop 2024-01-21T20:08:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.660+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 msgr2=0x7f315c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.660+0000 7f3162bcb700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3150013510 tx=0x7f315000b040).stop 2024-01-21T20:08:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.661+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 shutdown_connections 2024-01-21T20:08:28.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.661+0000 7f3162bcb700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f314c02ff60 0x7f314c032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:28.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.661+0000 7f3162bcb700 1 --2- 172.21.15.153:0/335733688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f315c0f6510 0x7f315c106730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:28.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.661+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 >> 172.21.15.153:0/335733688 conn(0x7f315c0f1c70 msgr2=0x7f315c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:28.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.661+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 shutdown_connections 2024-01-21T20:08:28.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:28.661+0000 7f3162bcb700 1 -- 172.21.15.153:0/335733688 wait complete. 2024-01-21T20:08:28.652 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:29.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:28 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:28.660532+0000 mon.smithi119 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.153:0/335733688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:30.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:30.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:30.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:29 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:28.736052+0000 mgr.smithi119.rqacbm (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:31.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:30 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:30.736411+0000 mgr.smithi119.rqacbm (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:31.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 -- 172.21.15.153:0/2519195368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 msgr2=0x7f67140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:31.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 --2- 172.21.15.153:0/2519195368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f67140f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f6708009fb0 tx=0x7f6708009b30).stop 2024-01-21T20:08:31.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 -- 172.21.15.153:0/2519195368 shutdown_connections 2024-01-21T20:08:31.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 --2- 172.21.15.153:0/2519195368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f67140f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 -- 172.21.15.153:0/2519195368 >> 172.21.15.153:0/2519195368 conn(0x7f67140f0b70 msgr2=0x7f67140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 -- 172.21.15.153:0/2519195368 shutdown_connections 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 -- 172.21.15.153:0/2519195368 wait complete. 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 Processor -- start 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.299+0000 7f671acfb700 1 -- start start 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f671acfb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f6714109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f671acfb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6714060240 con 0x7f67140f5410 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f6718a97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f6714109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f6718a97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f6714109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:37976/0 (socket says 172.21.15.153:37976) 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f6718a97700 1 -- 172.21.15.153:0/2574112758 learned_addr learned my addr 172.21.15.153:0/2574112758 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f6718a97700 1 -- 172.21.15.153:0/2574112758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708005040 con 0x7f67140f5410 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f6718a97700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f6714109430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f6708013510 tx=0x7f670800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6708009320 con 0x7f67140f5410 2024-01-21T20:08:31.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6708013f00 con 0x7f67140f5410 2024-01-21T20:08:31.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6708014d90 con 0x7f67140f5410 2024-01-21T20:08:31.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.300+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6714060440 con 0x7f67140f5410 2024-01-21T20:08:31.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.301+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6714109ae0 con 0x7f67140f5410 2024-01-21T20:08:31.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.301+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67080095f0 con 0x7f67140f5410 2024-01-21T20:08:31.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.301+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6714045ad0 con 0x7f67140f5410 2024-01-21T20:08:31.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.302+0000 7f67117fa700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f670402ff60 0x7f6704032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:31.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.302+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6708041000 con 0x7f67140f5410 2024-01-21T20:08:31.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.302+0000 7f6713fff700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f670402ff60 0x7f6704032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:31.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.302+0000 7f6713fff700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f670402ff60 0x7f6704032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f67000096a0 tx=0x7f6700006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:31.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.306+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f670801d7f0 con 0x7f67140f5410 2024-01-21T20:08:31.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.507+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67140fae60 con 0x7f67140f5410 2024-01-21T20:08:31.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.507+0000 7f67117fa700 1 -- 172.21.15.153:0/2574112758 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6708010230 con 0x7f67140f5410 2024-01-21T20:08:31.498 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:31.498 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f670402ff60 msgr2=0x7f6704032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f670402ff60 0x7f6704032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f67000096a0 tx=0x7f6700006b40).stop 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 msgr2=0x7f6714109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f6714109430 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f6708013510 tx=0x7f670800b040).stop 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 shutdown_connections 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f670402ff60 0x7f6704032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 --2- 172.21.15.153:0/2574112758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67140f5410 0x7f6714109430 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 >> 172.21.15.153:0/2574112758 conn(0x7f67140f0b70 msgr2=0x7f67140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 shutdown_connections 2024-01-21T20:08:31.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:31.509+0000 7f671acfb700 1 -- 172.21.15.153:0/2574112758 wait complete. 2024-01-21T20:08:31.501 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:32.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:32 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:31.509092+0000 mon.smithi119 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.153:0/2574112758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:32.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:32.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:33.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:33 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:32.736757+0000 mgr.smithi119.rqacbm (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.071+0000 7fa117964700 1 -- 172.21.15.153:0/2164000557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 msgr2=0x7fa1100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.071+0000 7fa117964700 1 --2- 172.21.15.153:0/2164000557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa1100f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fa100009fb0 tx=0x7fa100009b30).stop 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 -- 172.21.15.153:0/2164000557 shutdown_connections 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 --2- 172.21.15.153:0/2164000557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa1100f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 -- 172.21.15.153:0/2164000557 >> 172.21.15.153:0/2164000557 conn(0x7fa1100f0b70 msgr2=0x7fa1100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 -- 172.21.15.153:0/2164000557 shutdown_connections 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 -- 172.21.15.153:0/2164000557 wait complete. 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 Processor -- start 2024-01-21T20:08:34.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.072+0000 7fa117964700 1 -- start start 2024-01-21T20:08:34.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa117964700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa110109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa117964700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa110060240 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa115700700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa110109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa115700700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa110109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43400/0 (socket says 172.21.15.153:43400) 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa115700700 1 -- 172.21.15.153:0/3272210725 learned_addr learned my addr 172.21.15.153:0/3272210725 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa115700700 1 -- 172.21.15.153:0/3272210725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100005040 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa115700700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa110109430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fa100013510 tx=0x7fa10000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa100009320 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa110060440 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa100013f00 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.073+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa100014d90 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.074+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa110109ae0 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.074+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1000095f0 con 0x7fa1100f5410 2024-01-21T20:08:34.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.074+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa110045ad0 con 0x7fa1100f5410 2024-01-21T20:08:34.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.075+0000 7fa105ffb700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0fc02ff60 0x7fa0fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:34.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.075+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa100040fe0 con 0x7fa1100f5410 2024-01-21T20:08:34.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.075+0000 7fa114eff700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0fc02ff60 0x7fa0fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:34.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.075+0000 7fa114eff700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0fc02ff60 0x7fa0fc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa10c0096a0 tx=0x7fa10c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:34.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.079+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa100014420 con 0x7fa1100f5410 2024-01-21T20:08:34.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.287+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1100fae60 con 0x7fa1100f5410 2024-01-21T20:08:34.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.288+0000 7fa105ffb700 1 -- 172.21.15.153:0/3272210725 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa100010230 con 0x7fa1100f5410 2024-01-21T20:08:34.278 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:34.279 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:34.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0fc02ff60 msgr2=0x7fa0fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:34.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0fc02ff60 0x7fa0fc032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa10c0096a0 tx=0x7fa10c006b40).stop 2024-01-21T20:08:34.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 msgr2=0x7fa110109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa110109430 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fa100013510 tx=0x7fa10000b040).stop 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 shutdown_connections 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fa0fc02ff60 0x7fa0fc032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 --2- 172.21.15.153:0/3272210725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1100f5410 0x7fa110109430 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 >> 172.21.15.153:0/3272210725 conn(0x7fa1100f0b70 msgr2=0x7fa1100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 shutdown_connections 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:34.290+0000 7fa117964700 1 -- 172.21.15.153:0/3272210725 wait complete. 2024-01-21T20:08:34.281 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:34.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:34 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:34.289439+0000 mon.smithi119 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.153:0/3272210725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:35.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:35 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:34.737052+0000 mgr.smithi119.rqacbm (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:35.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:35.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 -- 172.21.15.153:0/3443438986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 msgr2=0x7fd25c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 --2- 172.21.15.153:0/3443438986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c0f8f20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd24c009fb0 tx=0x7fd24c009b30).stop 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 -- 172.21.15.153:0/3443438986 shutdown_connections 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 --2- 172.21.15.153:0/3443438986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c0f8f20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 -- 172.21.15.153:0/3443438986 >> 172.21.15.153:0/3443438986 conn(0x7fd25c0f09a0 msgr2=0x7fd25c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 -- 172.21.15.153:0/3443438986 shutdown_connections 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 -- 172.21.15.153:0/3443438986 wait complete. 2024-01-21T20:08:36.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 Processor -- start 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.837+0000 7fd25bfff700 1 -- start start 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25bfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c1069f0 con 0x7fd25c0601e0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43408/0 (socket says 172.21.15.153:43408) 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25affd700 1 -- 172.21.15.153:0/2547412223 learned_addr learned my addr 172.21.15.153:0/2547412223 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25affd700 1 -- 172.21.15.153:0/2547412223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c005040 con 0x7fd25c0601e0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25affd700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd24c013510 tx=0x7fd24c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd24c009320 con 0x7fd25c0601e0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25c106bf0 con 0x7fd25c0601e0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd24c013f00 con 0x7fd25c0601e0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd24c014d90 con 0x7fd25c0601e0 2024-01-21T20:08:36.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.838+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25c107080 con 0x7fd25c0601e0 2024-01-21T20:08:36.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.839+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd24c0263a0 con 0x7fd25c0601e0 2024-01-21T20:08:36.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.839+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25c045ad0 con 0x7fd25c0601e0 2024-01-21T20:08:36.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.839+0000 7fd2437fe700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd24402ff60 0x7fd244032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:36.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.839+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd24c0412f0 con 0x7fd25c0601e0 2024-01-21T20:08:36.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.840+0000 7fd25a7fc700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd24402ff60 0x7fd244032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:36.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.840+0000 7fd25a7fc700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd24402ff60 0x7fd244032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fd2500096a0 tx=0x7fd250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:36.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:36.844+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd24c0126f0 con 0x7fd25c0601e0 2024-01-21T20:08:37.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.050+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd25c107f30 con 0x7fd25c0601e0 2024-01-21T20:08:37.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.050+0000 7fd2437fe700 1 -- 172.21.15.153:0/2547412223 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd24c012330 con 0x7fd25c0601e0 2024-01-21T20:08:37.041 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:37.041 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:37.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd24402ff60 msgr2=0x7fd244032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd24402ff60 0x7fd244032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fd2500096a0 tx=0x7fd250006b40).stop 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 msgr2=0x7fd25c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c106500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd24c013510 tx=0x7fd24c00b040).stop 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 shutdown_connections 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fd24402ff60 0x7fd244032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 --2- 172.21.15.153:0/2547412223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25c0601e0 0x7fd25c106500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 >> 172.21.15.153:0/2547412223 conn(0x7fd25c0f09a0 msgr2=0x7fd25c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 shutdown_connections 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:37.052+0000 7fd25bfff700 1 -- 172.21.15.153:0/2547412223 wait complete. 2024-01-21T20:08:37.043 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:37.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:36.737344+0000 mgr.smithi119.rqacbm (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:37.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:37 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:37.051956+0000 mon.smithi119 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.153:0/2547412223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:38.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:38.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:39.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:38 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:38.737696+0000 mgr.smithi119.rqacbm (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:39.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.635+0000 7f09dfda8700 1 -- 172.21.15.153:0/1156612583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 msgr2=0x7f09d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.635+0000 7f09dfda8700 1 --2- 172.21.15.153:0/1156612583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d80f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f09cc009fb0 tx=0x7f09cc009b30).stop 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.635+0000 7f09dfda8700 1 -- 172.21.15.153:0/1156612583 shutdown_connections 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.635+0000 7f09dfda8700 1 --2- 172.21.15.153:0/1156612583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d80f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.635+0000 7f09dfda8700 1 -- 172.21.15.153:0/1156612583 >> 172.21.15.153:0/1156612583 conn(0x7f09d80f0b70 msgr2=0x7f09d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.636+0000 7f09dfda8700 1 -- 172.21.15.153:0/1156612583 shutdown_connections 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.636+0000 7f09dfda8700 1 -- 172.21.15.153:0/1156612583 wait complete. 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.636+0000 7f09dfda8700 1 Processor -- start 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.636+0000 7f09dfda8700 1 -- start start 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.636+0000 7f09dfda8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:39.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.636+0000 7f09dfda8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d8107240 con 0x7f09d80f5410 2024-01-21T20:08:39.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ddb44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:39.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ddb44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43424/0 (socket says 172.21.15.153:43424) 2024-01-21T20:08:39.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ddb44700 1 -- 172.21.15.153:0/2153414066 learned_addr learned my addr 172.21.15.153:0/2153414066 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:39.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ddb44700 1 -- 172.21.15.153:0/2153414066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc005040 con 0x7f09d80f5410 2024-01-21T20:08:39.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ddb44700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d810b850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f09cc013510 tx=0x7f09cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:39.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc009320 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09cc013f00 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09cc014d90 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d8060220 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.637+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d810bd40 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.638+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09cc0095f0 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.639+0000 7f09ca7fc700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f09c402ff60 0x7f09c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.639+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09cc0410c0 con 0x7f09d80f5410 2024-01-21T20:08:39.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.639+0000 7f09dd343700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f09c402ff60 0x7f09c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:39.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.639+0000 7f09dd343700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f09c402ff60 0x7f09c4032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f09d40096a0 tx=0x7f09d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:39.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.640+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d8059100 con 0x7f09d80f5410 2024-01-21T20:08:39.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.644+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09cc01d7f0 con 0x7f09d80f5410 2024-01-21T20:08:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.852+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09d8045ad0 con 0x7f09d80f5410 2024-01-21T20:08:39.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.853+0000 7f09ca7fc700 1 -- 172.21.15.153:0/2153414066 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09cc02b020 con 0x7f09d80f5410 2024-01-21T20:08:39.843 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:39.844 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f09c402ff60 msgr2=0x7f09c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f09c402ff60 0x7f09c4032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f09d40096a0 tx=0x7f09d4006b40).stop 2024-01-21T20:08:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 msgr2=0x7f09d810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d810b850 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f09cc013510 tx=0x7f09cc00b040).stop 2024-01-21T20:08:39.845 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 shutdown_connections 2024-01-21T20:08:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f09c402ff60 0x7f09c4032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 --2- 172.21.15.153:0/2153414066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09d80f5410 0x7f09d810b850 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.854+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 >> 172.21.15.153:0/2153414066 conn(0x7f09d80f0b70 msgr2=0x7f09d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.855+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 shutdown_connections 2024-01-21T20:08:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:39.855+0000 7f09dfda8700 1 -- 172.21.15.153:0/2153414066 wait complete. 2024-01-21T20:08:39.846 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:40.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:39 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:39.854242+0000 mon.smithi119 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.153:0/2153414066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:41.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:41.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:41.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:40 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:40.737992+0000 mgr.smithi119.rqacbm (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/737809345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 msgr2=0x7f3e840f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:42.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 --2- 172.21.15.153:0/737809345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e840f9190 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f3e74009fb0 tx=0x7f3e74009b30).stop 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/737809345 shutdown_connections 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 --2- 172.21.15.153:0/737809345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e840f9190 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/737809345 >> 172.21.15.153:0/737809345 conn(0x7f3e840f0b70 msgr2=0x7f3e840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/737809345 shutdown_connections 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.424+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/737809345 wait complete. 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e8bfbe700 1 Processor -- start 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e8bfbe700 1 -- start start 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e8bfbe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:42.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e8bfbe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84106bd0 con 0x7f3e840601e0 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e89d5a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e89d5a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:43438/0 (socket says 172.21.15.153:43438) 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.425+0000 7f3e89d5a700 1 -- 172.21.15.153:0/2199801160 learned_addr learned my addr 172.21.15.153:0/2199801160 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e89d5a700 1 -- 172.21.15.153:0/2199801160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74005040 con 0x7f3e840601e0 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e89d5a700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e841066e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f3e74004dd0 tx=0x7f3e74009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e74018070 con 0x7f3e840601e0 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e84106dd0 con 0x7f3e840601e0 2024-01-21T20:08:42.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e74013410 con 0x7f3e840601e0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e7401c3c0 con 0x7f3e840601e0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.426+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e84107260 con 0x7f3e840601e0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.427+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e7401c630 con 0x7f3e840601e0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.427+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e84045ad0 con 0x7f3e840601e0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.427+0000 7f3e7a7fc700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3e7002ff60 0x7f3e70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.427+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e740416f0 con 0x7f3e840601e0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.427+0000 7f3e89559700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3e7002ff60 0x7f3e70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:42.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.428+0000 7f3e89559700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3e7002ff60 0x7f3e70032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3e800096a0 tx=0x7f3e80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:42.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.432+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e7401ca30 con 0x7f3e840601e0 2024-01-21T20:08:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.640+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e841080b0 con 0x7f3e840601e0 2024-01-21T20:08:42.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.641+0000 7f3e7a7fc700 1 -- 172.21.15.153:0/2199801160 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3e7401b7e0 con 0x7f3e840601e0 2024-01-21T20:08:42.631 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:42.631 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3e7002ff60 msgr2=0x7f3e70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3e7002ff60 0x7f3e70032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3e800096a0 tx=0x7f3e80006b40).stop 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 msgr2=0x7f3e841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e841066e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f3e74004dd0 tx=0x7f3e74009b30).stop 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 shutdown_connections 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f3e7002ff60 0x7f3e70032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 --2- 172.21.15.153:0/2199801160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e840601e0 0x7f3e841066e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 >> 172.21.15.153:0/2199801160 conn(0x7f3e840f0b70 msgr2=0x7f3e840f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 shutdown_connections 2024-01-21T20:08:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:42.642+0000 7f3e8bfbe700 1 -- 172.21.15.153:0/2199801160 wait complete. 2024-01-21T20:08:42.634 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:43.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:42 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:42.642277+0000 mon.smithi119 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.153:0/2199801160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:44.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:44.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:44.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:43 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:42.738371+0000 mgr.smithi119.rqacbm (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:45.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:44 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:44.738630+0000 mgr.smithi119.rqacbm (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:45.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 -- 172.21.15.153:0/3585000263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 msgr2=0x7fe0840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 --2- 172.21.15.153:0/3585000263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe0840f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fe078009fb0 tx=0x7fe078009b30).stop 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 -- 172.21.15.153:0/3585000263 shutdown_connections 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 --2- 172.21.15.153:0/3585000263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe0840f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 -- 172.21.15.153:0/3585000263 >> 172.21.15.153:0/3585000263 conn(0x7fe0840f0b70 msgr2=0x7fe0840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 -- 172.21.15.153:0/3585000263 shutdown_connections 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.208+0000 7fe08bfcc700 1 -- 172.21.15.153:0/3585000263 wait complete. 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe08bfcc700 1 Processor -- start 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe08bfcc700 1 -- start start 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe08bfcc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe084105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:45.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe08bfcc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe084105b20 con 0x7fe0840f5410 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe089d68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe084105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe089d68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe084105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38036/0 (socket says 172.21.15.153:38036) 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.209+0000 7fe089d68700 1 -- 172.21.15.153:0/605983844 learned_addr learned my addr 172.21.15.153:0/605983844 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe089d68700 1 -- 172.21.15.153:0/605983844 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078005040 con 0x7fe0840f5410 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe089d68700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe084105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe078013510 tx=0x7fe07800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe078009320 con 0x7fe0840f5410 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe084105d20 con 0x7fe0840f5410 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe078013f00 con 0x7fe0840f5410 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe078014d90 con 0x7fe0840f5410 2024-01-21T20:08:45.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.210+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0841061b0 con 0x7fe0840f5410 2024-01-21T20:08:45.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.211+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0780095f0 con 0x7fe0840f5410 2024-01-21T20:08:45.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.211+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe084045ad0 con 0x7fe0840f5410 2024-01-21T20:08:45.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.211+0000 7fe0767fc700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe07002ff60 0x7fe070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:45.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.211+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe078041000 con 0x7fe0840f5410 2024-01-21T20:08:45.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.211+0000 7fe089567700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe07002ff60 0x7fe070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:45.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.212+0000 7fe089567700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe07002ff60 0x7fe070032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:45.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.215+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe078014420 con 0x7fe0840f5410 2024-01-21T20:08:45.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.424+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0840fae60 con 0x7fe0840f5410 2024-01-21T20:08:45.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.425+0000 7fe0767fc700 1 -- 172.21.15.153:0/605983844 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe078010230 con 0x7fe0840f5410 2024-01-21T20:08:45.415 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:45.415 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe07002ff60 msgr2=0x7fe070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe07002ff60 0x7fe070032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe0800096a0 tx=0x7fe080006b40).stop 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 msgr2=0x7fe084105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe084105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe078013510 tx=0x7fe07800b040).stop 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 shutdown_connections 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fe07002ff60 0x7fe070032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 --2- 172.21.15.153:0/605983844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0840f5410 0x7fe084105630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.426+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 >> 172.21.15.153:0/605983844 conn(0x7fe0840f0b70 msgr2=0x7fe0840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:45.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.427+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 shutdown_connections 2024-01-21T20:08:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:45.427+0000 7fe08bfcc700 1 -- 172.21.15.153:0/605983844 wait complete. 2024-01-21T20:08:45.418 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:46.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:45 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:45.426188+0000 mon.smithi119 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.153:0/605983844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:46.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:46.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:47.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:47 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:46.738912+0000 mgr.smithi119.rqacbm (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 -- 172.21.15.153:0/1530552919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 msgr2=0x7fba980f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 --2- 172.21.15.153:0/1530552919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba980f4f20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fba88009fb0 tx=0x7fba88009b30).stop 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 -- 172.21.15.153:0/1530552919 shutdown_connections 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 --2- 172.21.15.153:0/1530552919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba980f4f20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 -- 172.21.15.153:0/1530552919 >> 172.21.15.153:0/1530552919 conn(0x7fba980f0260 msgr2=0x7fba980f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 -- 172.21.15.153:0/1530552919 shutdown_connections 2024-01-21T20:08:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.016+0000 7fbaa018b700 1 -- 172.21.15.153:0/1530552919 wait complete. 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fbaa018b700 1 Processor -- start 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fbaa018b700 1 -- start start 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fbaa018b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba98106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fbaa018b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba98106bf0 con 0x7fba980f4b00 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fba9df27700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba98106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fba9df27700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba98106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38068/0 (socket says 172.21.15.153:38068) 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.017+0000 7fba9df27700 1 -- 172.21.15.153:0/88238465 learned_addr learned my addr 172.21.15.153:0/88238465 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fba9df27700 1 -- 172.21.15.153:0/88238465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88005040 con 0x7fba980f4b00 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fba9df27700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba98106700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fba88013510 tx=0x7fba8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba88009320 con 0x7fba980f4b00 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba98106df0 con 0x7fba980f4b00 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba88013f00 con 0x7fba980f4b00 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba88014d90 con 0x7fba980f4b00 2024-01-21T20:08:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.018+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba98107280 con 0x7fba980f4b00 2024-01-21T20:08:48.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.019+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba88014420 con 0x7fba980f4b00 2024-01-21T20:08:48.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.019+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98045ad0 con 0x7fba980f4b00 2024-01-21T20:08:48.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.019+0000 7fba8e7fc700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba8402ff60 0x7fba84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:48.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.019+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba88041000 con 0x7fba980f4b00 2024-01-21T20:08:48.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.019+0000 7fba9d726700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba8402ff60 0x7fba84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:48.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.020+0000 7fba9d726700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba8402ff60 0x7fba84032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fba940096a0 tx=0x7fba94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:48.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.024+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba880095f0 con 0x7fba980f4b00 2024-01-21T20:08:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.227+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba980fa5c0 con 0x7fba980f4b00 2024-01-21T20:08:48.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.228+0000 7fba8e7fc700 1 -- 172.21.15.153:0/88238465 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba8801db70 con 0x7fba980f4b00 2024-01-21T20:08:48.218 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:48.218 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba8402ff60 msgr2=0x7fba84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba8402ff60 0x7fba84032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fba940096a0 tx=0x7fba94006b40).stop 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 msgr2=0x7fba98106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba98106700 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7fba88013510 tx=0x7fba8800b040).stop 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 shutdown_connections 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7fba8402ff60 0x7fba84032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 --2- 172.21.15.153:0/88238465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba980f4b00 0x7fba98106700 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 >> 172.21.15.153:0/88238465 conn(0x7fba980f0260 msgr2=0x7fba980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 shutdown_connections 2024-01-21T20:08:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:48.229+0000 7fbaa018b700 1 -- 172.21.15.153:0/88238465 wait complete. 2024-01-21T20:08:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:48.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:48 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:48.229203+0000 mon.smithi119 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.153:0/88238465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:49.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:49 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:48.739200+0000 mgr.smithi119.rqacbm (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:49.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:49.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:50.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 -- 172.21.15.153:0/2870263569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 msgr2=0x7f75480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 --2- 172.21.15.153:0/2870263569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f75480f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f7538009fb0 tx=0x7f7538009b30).stop 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 -- 172.21.15.153:0/2870263569 shutdown_connections 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 --2- 172.21.15.153:0/2870263569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f75480f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 -- 172.21.15.153:0/2870263569 >> 172.21.15.153:0/2870263569 conn(0x7f75480f0b70 msgr2=0x7f75480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 -- 172.21.15.153:0/2870263569 shutdown_connections 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.837+0000 7f754e694700 1 -- 172.21.15.153:0/2870263569 wait complete. 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f754e694700 1 Processor -- start 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f754e694700 1 -- start start 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f754e694700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f7548106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f754e694700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7548106c10 con 0x7f75480f5410 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f7547fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f7548106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:50.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f7547fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f7548106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:38090/0 (socket says 172.21.15.153:38090) 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.838+0000 7f7547fff700 1 -- 172.21.15.153:0/3669435168 learned_addr learned my addr 172.21.15.153:0/3669435168 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f7547fff700 1 -- 172.21.15.153:0/3669435168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7538005040 con 0x7f75480f5410 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f7547fff700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f7548106720 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7538014000 tx=0x7f753800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75380095f0 con 0x7f75480f5410 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7538007e70 con 0x7f75480f5410 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7548106e10 con 0x7f75480f5410 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7538014ea0 con 0x7f75480f5410 2024-01-21T20:08:50.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.839+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75481072a0 con 0x7f75480f5410 2024-01-21T20:08:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.840+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7538025070 con 0x7f75480f5410 2024-01-21T20:08:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.840+0000 7f7544ff9700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f753002ffb0 0x7f7530032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.840+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7548045ad0 con 0x7f75480f5410 2024-01-21T20:08:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.840+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75380409a0 con 0x7f75480f5410 2024-01-21T20:08:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.840+0000 7f75477fe700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f753002ffb0 0x7f7530032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:50.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.840+0000 7f75477fe700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f753002ffb0 0x7f7530032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f753c0096a0 tx=0x7f753c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:50.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:50.844+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f753800f4e0 con 0x7f75480f5410 2024-01-21T20:08:51.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.051+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7548108150 con 0x7f75480f5410 2024-01-21T20:08:51.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.053+0000 7f7544ff9700 1 -- 172.21.15.153:0/3669435168 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f753800f300 con 0x7f75480f5410 2024-01-21T20:08:51.043 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:51.043 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:51.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f753002ffb0 msgr2=0x7f7530032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:51.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f753002ffb0 0x7f7530032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f753c0096a0 tx=0x7f753c006b40).stop 2024-01-21T20:08:51.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 msgr2=0x7f7548106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:51.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f7548106720 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7538014000 tx=0x7f753800b040).stop 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 shutdown_connections 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f753002ffb0 0x7f7530032470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 --2- 172.21.15.153:0/3669435168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75480f5410 0x7f7548106720 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 >> 172.21.15.153:0/3669435168 conn(0x7f75480f0b70 msgr2=0x7f754805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 shutdown_connections 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:51.054+0000 7f754e694700 1 -- 172.21.15.153:0/3669435168 wait complete. 2024-01-21T20:08:51.045 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:51.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:50 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:50.739469+0000 mgr.smithi119.rqacbm (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:52.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:51 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:51.053425+0000 mon.smithi119 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.153:0/3669435168' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:52.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:52.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:53.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:52 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:52.739825+0000 mgr.smithi119.rqacbm (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:53.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.677+0000 7f9fb2acd700 1 -- 172.21.15.153:0/86597408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 msgr2=0x7f9fac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.677+0000 7f9fb2acd700 1 --2- 172.21.15.153:0/86597408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac0f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9fa0009fb0 tx=0x7f9fa0009b30).stop 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.677+0000 7f9fb2acd700 1 -- 172.21.15.153:0/86597408 shutdown_connections 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.677+0000 7f9fb2acd700 1 --2- 172.21.15.153:0/86597408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac0f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 -- 172.21.15.153:0/86597408 >> 172.21.15.153:0/86597408 conn(0x7f9fac0f1c70 msgr2=0x7f9fac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 -- 172.21.15.153:0/86597408 shutdown_connections 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 -- 172.21.15.153:0/86597408 wait complete. 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 Processor -- start 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 -- start start 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.678+0000 7f9fb2acd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac106bc0 con 0x7f9fac0f6510 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb0869700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb0869700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39172/0 (socket says 172.21.15.153:39172) 2024-01-21T20:08:53.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb0869700 1 -- 172.21.15.153:0/1802732678 learned_addr learned my addr 172.21.15.153:0/1802732678 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb0869700 1 -- 172.21.15.153:0/1802732678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa0005040 con 0x7f9fac0f6510 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb0869700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac1066d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9fa0013510 tx=0x7f9fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa0009320 con 0x7f9fac0f6510 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac106dc0 con 0x7f9fac0f6510 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fa0013f00 con 0x7f9fac0f6510 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa0014d90 con 0x7f9fac0f6510 2024-01-21T20:08:53.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.679+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fac107250 con 0x7f9fac0f6510 2024-01-21T20:08:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.680+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fa0014420 con 0x7f9fac0f6510 2024-01-21T20:08:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.680+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac045ad0 con 0x7f9fac0f6510 2024-01-21T20:08:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.680+0000 7f9fa97fa700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9f9c02ffb0 0x7f9f9c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.680+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fa00410c0 con 0x7f9fac0f6510 2024-01-21T20:08:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.680+0000 7f9fabfff700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9f9c02ffb0 0x7f9f9c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.681+0000 7f9fabfff700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9f9c02ffb0 0x7f9f9c032470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9f980096a0 tx=0x7f9f98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:53.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.685+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fa00095f0 con 0x7f9fac0f6510 2024-01-21T20:08:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.890+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fac108100 con 0x7f9fac0f6510 2024-01-21T20:08:53.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.891+0000 7f9fa97fa700 1 -- 172.21.15.153:0/1802732678 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9fa001db70 con 0x7f9fac0f6510 2024-01-21T20:08:53.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:53.882 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9f9c02ffb0 msgr2=0x7f9f9c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9f9c02ffb0 0x7f9f9c032470 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9f980096a0 tx=0x7f9f98006b40).stop 2024-01-21T20:08:53.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 msgr2=0x7f9fac1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac1066d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9fa0013510 tx=0x7f9fa000b040).stop 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 shutdown_connections 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f9f9c02ffb0 0x7f9f9c032470 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 --2- 172.21.15.153:0/1802732678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fac0f6510 0x7f9fac1066d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 >> 172.21.15.153:0/1802732678 conn(0x7f9fac0f1c70 msgr2=0x7f9fac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 shutdown_connections 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:53.893+0000 7f9fb2acd700 1 -- 172.21.15.153:0/1802732678 wait complete. 2024-01-21T20:08:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:54.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:53 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:53.892624+0000 mon.smithi119 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.153:0/1802732678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:55.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:55.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:55.336 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:54 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:54.740199+0000 mgr.smithi119.rqacbm (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 -- 172.21.15.153:0/3183742317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 msgr2=0x7f39780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 --2- 172.21.15.153:0/3183742317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f39780f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f3968009fb0 tx=0x7f3968009b30).stop 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 -- 172.21.15.153:0/3183742317 shutdown_connections 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 --2- 172.21.15.153:0/3183742317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f39780f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 -- 172.21.15.153:0/3183742317 >> 172.21.15.153:0/3183742317 conn(0x7f39780f0b70 msgr2=0x7f39780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 -- 172.21.15.153:0/3183742317 shutdown_connections 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.472+0000 7f397e11d700 1 -- 172.21.15.153:0/3183742317 wait complete. 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.473+0000 7f397e11d700 1 Processor -- start 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.473+0000 7f397e11d700 1 -- start start 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.473+0000 7f397e11d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f3978105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.473+0000 7f397e11d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3978105b20 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f39777fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f3978105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f39777fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f3978105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39194/0 (socket says 172.21.15.153:39194) 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f39777fe700 1 -- 172.21.15.153:0/4200329092 learned_addr learned my addr 172.21.15.153:0/4200329092 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f39777fe700 1 -- 172.21.15.153:0/4200329092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968005040 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f39777fe700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f3978105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f3968013510 tx=0x7f396800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3968009320 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3968013f00 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3978105d20 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3968014d90 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.474+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39781061b0 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.475+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f396800f410 con 0x7f39780f5410 2024-01-21T20:08:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.475+0000 7f395ffff700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f396002ff60 0x7f3960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:56.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.475+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3978045ad0 con 0x7f39780f5410 2024-01-21T20:08:56.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.476+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3968042e60 con 0x7f39780f5410 2024-01-21T20:08:56.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.476+0000 7f3976ffd700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f396002ff60 0x7f3960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:56.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.476+0000 7f3976ffd700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f396002ff60 0x7f3960032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f396c0096a0 tx=0x7f396c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.480+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39680126f0 con 0x7f39780f5410 2024-01-21T20:08:56.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.684+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39780faf50 con 0x7f39780f5410 2024-01-21T20:08:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.684+0000 7f395ffff700 1 -- 172.21.15.153:0/4200329092 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3968012330 con 0x7f39780f5410 2024-01-21T20:08:56.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:56.675 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.686+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f396002ff60 msgr2=0x7f3960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.686+0000 7f397e11d700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f396002ff60 0x7f3960032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f396c0096a0 tx=0x7f396c006b40).stop 2024-01-21T20:08:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.686+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 msgr2=0x7f3978105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:56.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.686+0000 7f397e11d700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f3978105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f3968013510 tx=0x7f396800b040).stop 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.687+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 shutdown_connections 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.687+0000 7f397e11d700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f396002ff60 0x7f3960032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.687+0000 7f397e11d700 1 --2- 172.21.15.153:0/4200329092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39780f5410 0x7f3978105630 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.687+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 >> 172.21.15.153:0/4200329092 conn(0x7f39780f0b70 msgr2=0x7f39780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.687+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 shutdown_connections 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:56.687+0000 7f397e11d700 1 -- 172.21.15.153:0/4200329092 wait complete. 2024-01-21T20:08:56.678 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:08:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:56.685861+0000 mon.smithi119 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.153:0/4200329092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:08:57.587 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:57 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:56.740586+0000 mgr.smithi119.rqacbm (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:58.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:08:58.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:08:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:08:58.740863+0000 mgr.smithi119.rqacbm (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:08:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:58.789211+0000 mon.smithi119 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/mirror_snapshot_schedule"}]: dispatch 2024-01-21T20:08:59.087 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:58 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:58.789429+0000 mon.smithi119 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.119:0/1392704063' entity='mgr.smithi119.rqacbm' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi119.rqacbm/trash_purge_schedule"}]: dispatch 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 -- 172.21.15.153:0/4248089112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 msgr2=0x7f95540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 --2- 172.21.15.153:0/4248089112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f95540f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f9548009fb0 tx=0x7f9548009b30).stop 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 -- 172.21.15.153:0/4248089112 shutdown_connections 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 --2- 172.21.15.153:0/4248089112 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f95540f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 -- 172.21.15.153:0/4248089112 >> 172.21.15.153:0/4248089112 conn(0x7f95540f0b70 msgr2=0x7f95540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 -- 172.21.15.153:0/4248089112 shutdown_connections 2024-01-21T20:08:59.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.280+0000 7f955af1d700 1 -- 172.21.15.153:0/4248089112 wait complete. 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.281+0000 7f955af1d700 1 Processor -- start 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.281+0000 7f955af1d700 1 -- start start 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.281+0000 7f955af1d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f9554105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.281+0000 7f955af1d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9554105b20 con 0x7f95540f5410 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f9558cb9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f9554105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f9558cb9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f9554105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39206/0 (socket says 172.21.15.153:39206) 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f9558cb9700 1 -- 172.21.15.153:0/254925059 learned_addr learned my addr 172.21.15.153:0/254925059 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f9558cb9700 1 -- 172.21.15.153:0/254925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9548005040 con 0x7f95540f5410 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f9558cb9700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f9554105630 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9548014000 tx=0x7f954800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95480095f0 con 0x7f95540f5410 2024-01-21T20:08:59.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9548007e70 con 0x7f95540f5410 2024-01-21T20:08:59.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9554105d20 con 0x7f95540f5410 2024-01-21T20:08:59.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9548014ea0 con 0x7f95540f5410 2024-01-21T20:08:59.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.282+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95541061b0 con 0x7f95540f5410 2024-01-21T20:08:59.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.283+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f954801d4d0 con 0x7f95540f5410 2024-01-21T20:08:59.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.283+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9554045ad0 con 0x7f95540f5410 2024-01-21T20:08:59.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.283+0000 7f95517fa700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f954402ffb0 0x7f9544032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:08:59.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.283+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9548040c00 con 0x7f95540f5410 2024-01-21T20:08:59.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.284+0000 7f9553fff700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f954402ffb0 0x7f9544032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:08:59.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.284+0000 7f9553fff700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f954402ffb0 0x7f9544032470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f95400096a0 tx=0x7f9540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:08:59.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.288+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f954800f710 con 0x7f95540f5410 2024-01-21T20:08:59.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.495+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95540fae60 con 0x7f95540f5410 2024-01-21T20:08:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.496+0000 7f95517fa700 1 -- 172.21.15.153:0/254925059 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95480252f0 con 0x7f95540f5410 2024-01-21T20:08:59.487 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:08:59.487 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:08:59.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f954402ffb0 msgr2=0x7f9544032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:59.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f954402ffb0 0x7f9544032470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f95400096a0 tx=0x7f9540006b40).stop 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 msgr2=0x7f9554105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f9554105630 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9548014000 tx=0x7f954800b040).stop 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 shutdown_connections 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f954402ffb0 0x7f9544032470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 --2- 172.21.15.153:0/254925059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95540f5410 0x7f9554105630 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 >> 172.21.15.153:0/254925059 conn(0x7f95540f0b70 msgr2=0x7f95540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 shutdown_connections 2024-01-21T20:08:59.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:08:59.498+0000 7f955af1d700 1 -- 172.21.15.153:0/254925059 wait complete. 2024-01-21T20:08:59.490 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:09:00.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:08:59 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:08:59.497658+0000 mon.smithi119 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.153:0/254925059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:09:00.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-21T20:09:00.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79b26e34-b897-11ee-95b0-87774f69a715 -- ceph mon dump -f json 2024-01-21T20:09:01.086 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:09:00 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: cluster 2024-01-21T20:09:00.741113+0000 mgr.smithi119.rqacbm (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-21T20:09:02.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- 172.21.15.153:0/3319736931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 msgr2=0x7f25900f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:09:02.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 --2- 172.21.15.153:0/3319736931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f25900f66c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f2580009fb0 tx=0x7f2580009b30).stop 2024-01-21T20:09:02.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- 172.21.15.153:0/3319736931 shutdown_connections 2024-01-21T20:09:02.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 --2- 172.21.15.153:0/3319736931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f25900f66c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:09:02.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- 172.21.15.153:0/3319736931 >> 172.21.15.153:0/3319736931 conn(0x7f25900f1aa0 msgr2=0x7f25900f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:09:02.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- 172.21.15.153:0/3319736931 shutdown_connections 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- 172.21.15.153:0/3319736931 wait complete. 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 Processor -- start 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- start start 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f259006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.064+0000 7f259549b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2590069e20 con 0x7f25900f62a0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f259006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f259006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.153:39212/0 (socket says 172.21.15.153:39212) 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258ffff700 1 -- 172.21.15.153:0/2360593807 learned_addr learned my addr 172.21.15.153:0/2360593807 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258ffff700 1 -- 172.21.15.153:0/2360593807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2580005040 con 0x7f25900f62a0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258ffff700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f259006b9a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f2580013510 tx=0x7f258000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2580009320 con 0x7f25900f62a0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2580013f00 con 0x7f25900f62a0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2590069fc0 con 0x7f25900f62a0 2024-01-21T20:09:02.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2580014dc0 con 0x7f25900f62a0 2024-01-21T20:09:02.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.065+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259006a450 con 0x7f25900f62a0 2024-01-21T20:09:02.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.066+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f258001d3d0 con 0x7f25900f62a0 2024-01-21T20:09:02.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.066+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2590059100 con 0x7f25900f62a0 2024-01-21T20:09:02.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.066+0000 7f258cff9700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-21T20:09:02.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.066+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25800410d0 con 0x7f25900f62a0 2024-01-21T20:09:02.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.066+0000 7f258f7fe700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-21T20:09:02.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.067+0000 7f258f7fe700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f25840096a0 tx=0x7f2584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T20:09:02.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.071+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25800095f0 con 0x7f25900f62a0 2024-01-21T20:09:02.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.278+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25900fbce0 con 0x7f25900f62a0 2024-01-21T20:09:02.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.278+0000 7f258cff9700 1 -- 172.21.15.153:0/2360593807 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f258001cb10 con 0x7f25900f62a0 2024-01-21T20:09:02.269 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-01-21T20:09:02.269 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"79b26e34-b897-11ee-95b0-87774f69a715","modified":"2024-01-21T19:59:06.037046Z","created":"2024-01-21T19:59:06.037046Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi119","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 msgr2=0x7f2578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f25840096a0 tx=0x7f2584006b40).stop 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 msgr2=0x7f259006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f259006b9a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f2580013510 tx=0x7f258000b040).stop 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 shutdown_connections 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:6800/1043201922,v1:172.21.15.119:6801/1043201922] conn(0x7f257802ff60 0x7f2578032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 --2- 172.21.15.153:0/2360593807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25900f62a0 0x7f259006b9a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 >> 172.21.15.153:0/2360593807 conn(0x7f25900f1aa0 msgr2=0x7f25900f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 shutdown_connections 2024-01-21T20:09:02.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-01-21T20:09:02.280+0000 7f259549b700 1 -- 172.21.15.153:0/2360593807 wait complete. 2024-01-21T20:09:02.272 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-01-21T20:09:02.586 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:09:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: audit 2024-01-21T20:09:02.279959+0000 mon.smithi119 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.153:0/2360593807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-21T20:09:02.683 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-21T20:09:02.684 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-21T20:09:02.685 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T20:09:02.708 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-21T20:09:02.727 INFO:tasks.cephadm:Stopping all daemons... 2024-01-21T20:09:02.727 INFO:tasks.cephadm.mon.smithi119:Stopping mon.smithi119... 2024-01-21T20:09:02.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119 2024-01-21T20:09:03.039 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:09:02 smithi119 systemd[1]: Stopping Ceph mon.smithi119 for 79b26e34-b897-11ee-95b0-87774f69a715... 2024-01-21T20:09:03.040 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:09:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T20:09:02.873+0000 7fec83f94700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi119 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-21T20:09:03.040 INFO:journalctl@ceph.mon.smithi119.smithi119.stdout:Jan 21 20:09:02 smithi119 ceph-79b26e34-b897-11ee-95b0-87774f69a715-mon.smithi119[38815]: debug 2024-01-21T20:09:02.873+0000 7fec83f94700 -1 mon.smithi119@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T20:09:03.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi119.service' 2024-01-21T20:09:03.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-21T20:09:03.223 INFO:tasks.cephadm.mon.smithi119:Stopped mon.smithi119 2024-01-21T20:09:03.223 INFO:tasks.cephadm.mon.smithi153:Stopping mon.smithi153... 2024-01-21T20:09:03.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153 2024-01-21T20:09:03.255 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153.service: Unit ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153.service not loaded. 2024-01-21T20:09:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-21T20:09:03.257 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi153" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 5: 'sudo systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153' 2024-01-21T20:09:03.258 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-21T20:09:03.259 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/remote/ubuntu@smithi119.front.sepia.ceph.com/crash 2024-01-21T20:09:03.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/crash -- . 2024-01-21T20:09:03.298 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/remote/ubuntu@smithi153.front.sepia.ceph.com/crash 2024-01-21T20:09:03.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/crash -- . 2024-01-21T20:09:03.346 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/79b26e34-b897-11ee-95b0-87774f69a715/crash: Cannot open: No such file or directory 2024-01-21T20:09:03.346 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-01-21T20:09:03.347 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-21T20:09:03.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-21T20:09:03.375 INFO:teuthology.orchestra.run.smithi119.stderr:grep: /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph.log: No such file or directory 2024-01-21T20:09:03.377 INFO:tasks.cephadm:Compressing logs... 2024-01-21T20:09:03.377 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T20:09:03.419 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T20:09:03.443 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T20:09:03.444 INFO:teuthology.orchestra.run.smithi119.stderr:: No such file or directory 2024-01-21T20:09:03.444 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-volume.log 2024-01-21T20:09:03.445 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mon.smithi119.log 2024-01-21T20:09:03.445 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi119.rqacbm.log 2024-01-21T20:09:03.445 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-volume.log: /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mon.smithi119.log: /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi119.rqacbm.log: 88.4% -- replaced with /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-volume.log.gz 2024-01-21T20:09:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-21T20:09:03.445 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-01-21T20:09:03.447 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T20:09:03.449 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-01-21T20:09:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m0.013s 2024-01-21T20:09:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m0.008s 2024-01-21T20:09:03.449 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.013s 2024-01-21T20:09:03.449 INFO:teuthology.orchestra.run.smithi119.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-21T20:09:03.522 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi119.rqacbm.log: file size changed while zipping 2024-01-21T20:09:03.522 INFO:teuthology.orchestra.run.smithi119.stderr: 92.8% -- replaced with /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mgr.smithi119.rqacbm.log.gz 2024-01-21T20:09:03.588 INFO:teuthology.orchestra.run.smithi119.stderr: 90.7% -- replaced with /var/log/ceph/79b26e34-b897-11ee-95b0-87774f69a715/ceph-mon.smithi119.log.gz 2024-01-21T20:09:03.589 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-01-21T20:09:03.589 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m0.155s 2024-01-21T20:09:03.589 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m0.226s 2024-01-21T20:09:03.589 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m0.020s 2024-01-21T20:09:03.590 INFO:tasks.cephadm:Archiving logs... 2024-01-21T20:09:03.590 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/remote/ubuntu@smithi119.front.sepia.ceph.com/log 2024-01-21T20:09:03.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T20:09:03.705 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/remote/ubuntu@smithi153.front.sepia.ceph.com/log 2024-01-21T20:09:03.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T20:09:03.733 INFO:tasks.cephadm:Removing cluster... 2024-01-21T20:09:03.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79b26e34-b897-11ee-95b0-87774f69a715 --force 2024-01-21T20:09:55.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 79b26e34-b897-11ee-95b0-87774f69a715 --force 2024-01-21T20:09:56.248 INFO:tasks.cephadm:Removing cephadm ... 2024-01-21T20:09:56.248 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T20:09:56.267 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-21T20:09:56.285 INFO:tasks.cephadm:Teardown complete 2024-01-21T20:09:56.285 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 5: 'sudo systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153' 2024-01-21T20:09:56.509 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=08cbfdc1acb24026b17f98a334682044 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_ea310589757b987e8ba9e9ba96dfa9a6f9c1e8ec/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 5: 'sudo systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153' 2024-01-21T20:09:56.514 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-21T20:09:56.525 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T20:09:56.554 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T20:09:56.554 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T20:09:56.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T20:09:56.646 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T20:09:56.655 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T20:09:56.655 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T20:09:56.658 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T20:09:56.673 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2024-01-21T20:09:56.675 INFO:teuthology.orchestra.run.smithi153.stderr:bash: ntpq: command not found 2024-01-21T20:09:56.679 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T20:09:56.679 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-01-21T20:09:56.679 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 2 6 377 21 -513us[ -499us] +/- 77ms 2024-01-21T20:09:56.679 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 -575us[ -561us] +/- 80ms 2024-01-21T20:09:56.679 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +947us[ +961us] +/- 65ms 2024-01-21T20:09:56.680 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T20:09:56.680 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T20:09:56.681 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-01-21T20:09:56.681 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 2 6 377 21 -522us[ -518us] +/- 77ms 2024-01-21T20:09:56.681 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -584us[ -580us] +/- 80ms 2024-01-21T20:09:56.681 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +956us[ +961us] +/- 65ms 2024-01-21T20:09:56.681 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T20:09:56.681 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T20:09:56.692 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T20:09:56.693 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T20:09:56.723 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T20:09:56.787 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T20:09:56.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T20:09:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:56.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T20:09:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:56.910 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T20:09:56.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T20:09:56.949 INFO:teuthology.task.internal:Duration was 1216.090164 seconds 2024-01-21T20:09:56.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T20:09:56.982 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T20:09:56.982 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T20:09:56.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T20:09:57.030 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T20:09:57.030 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T20:09:57.346 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T20:09:57.346 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-01-21T20:09:57.346 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T20:09:57.368 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-01-21T20:09:57.368 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T20:09:57.391 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T20:09:57.392 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T20:09:57.410 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T20:09:57.481 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T20:09:57.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T20:09:57.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T20:09:57.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T20:09:57.581 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T20:09:57.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T20:09:57.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T20:09:57.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T20:09:57.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:57.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:57.674 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-01-21T20:09:57.701 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-01-21T20:09:57.731 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:57.765 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T20:09:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T20:09:57.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T20:09:57.798 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T20:09:57.798 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/remote/smithi119 2024-01-21T20:09:57.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T20:09:57.844 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071/remote/smithi153 2024-01-21T20:09:57.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T20:09:57.883 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T20:09:57.883 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T20:09:57.885 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T20:09:57.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T20:09:57.947 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T20:09:57.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T20:09:57.975 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T20:09:57.975 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T20:09:57.977 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T20:09:57.992 INFO:teuthology.orchestra.run.smithi119.stdout: 525353 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 20:09 /home/ubuntu/cephtest 2024-01-21T20:09:57.994 INFO:teuthology.orchestra.run.smithi153.stdout: 525360 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 20:09 /home/ubuntu/cephtest 2024-01-21T20:09:57.996 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T20:09:58.009 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T20:09:58.075 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T20:09:58.101 DEBUG:teuthology.nuke:shortname: smithi119 2024-01-21T20:09:58.102 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T20:09:58.124 DEBUG:teuthology.nuke:shortname: smithi153 2024-01-21T20:09:58.124 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T20:09:58.128 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:41:23.004969', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJW5n2c9TyhtohRmtZgkhPVMm5LGgNPhxEtr2nFYL00v9GCcP28JflQFozomYWCYHu4RaVmNbH+VVgYG6DCzMg0='} 2024-01-21T20:09:58.145 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-21_16:37:18-fs-wip-yuri4-testing-2024-01-18-1257-pacific-distro-default-smithi/7525071', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 19:41:23.006297', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCRPqxZJRPPOqbJwp7GTJTBwgO6jxWOJIP+JCFihE9/i6l3IzUlgxoW4pzVtvM67bT9cuerqEvK/kb266Kd38vg='} 2024-01-21T20:09:58.178 INFO:teuthology.orchestra.console.smithi119:Power off 2024-01-21T20:09:58.178 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T20:09:58.192 INFO:teuthology.orchestra.console.smithi153:Power off 2024-01-21T20:09:58.193 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T20:09:58.204 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-01-21T20:09:58.204 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:58.215 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-01-21T20:09:58.215 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:09:58.227 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-01-21T20:09:58.236 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-01-21T20:10:02.229 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:10:02.238 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:10:02.340 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-01-21T20:10:02.347 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-01-21T20:10:06.344 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:10:06.353 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:10:06.455 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-01-21T20:10:06.556 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-01-21T20:10:06.657 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-01-21T20:10:10.560 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T20:10:10.669 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-01-21T20:10:10.770 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-01-21T20:10:10.875 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1216.090164422989 failure_reason: 'Command failed on smithi153 with status 5: ''sudo systemctl stop ceph-79b26e34-b897-11ee-95b0-87774f69a715@mon.smithi153''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=08cbfdc1acb24026b17f98a334682044 status: fail success: false 2024-01-21T20:10:10.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T20:10:10.947 INFO:teuthology.run:FAIL